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How and Why This Program Evaluation 
Handbook Was Developed 



R ecruiting New Teachers, Inc. (RNT) 
embarked on the program evaluation 
handbook project in 1996 in response to 
numerous requests for help from precollegiate 
teacher recruitment program coordinators. Our 
pathbreaking research into precollegiate teacher 
recruitment programs in 1992 and again in 
1994 had led to the publication of two 
definitive reports, Teachings Next Generation and 
Teaching's Next Generation: Five Years On and 
Growing and the establishment of a 
clearinghouse at RNT called the National 
Center on Precollegiate Teacher Recruitment. 
Under the Centers auspices, we learned via 
national surveys, conferences and calls for 
technical assistance that program coordinators 
were highly dissatisfied with their ability to 
evaluate program outcomes and effectiveness. 
Many recognized the need for more systematic 
means of formative evaluation to drive 
continuous program improvement. 

Coordinators decried the absence of practical, 
reliable evaluation tools and the lack of time, 
resources, and wider institutional support that 
made it almost impossible to track program 
graduates along pathways into teaching careers. 
Most recognized that careful evaluation is 
essential to inform decision making and justify 
continued investment in and support for their 
programs. 

Twice we convened focus groups of 
practitioners representing a variety of teacher 
recruitment program models — statewide, inner- 
city, and national in scope — who had all 
commissioned external evaluation and tracking 
studies. Three program evaluation experts — 

Dr. Barnett Berry, Dr. Beatriz Chu Clewell, 
and Dr. Shannon Cahill — also participated 
actively in our discussions and planning. All 
agreed that recruitment program coordinators 
have a critical need for step-by-step guidance 
on examining the effectiveness of program 
activities and strategies and on tracking program 
participants’ progress after graduation. 



The end product, Measured Steps: An 
Evaluation Handbook for Improving Teacher 
Recruitment Programs, has been designed for the 
beginning evaluator. We deleted “precollegiate” 
from the title because feedback on our first draft 
convinced us of the handbooks relevance and 
potential benefits to all teacher recruitment 
program directors. 

Taking a “hands-on” approach, we have 
included many worksheets and instructions for 
using them. While a few other practical and 
user-friendly guides to program evaluation are 
available (see the Resources section on page 97), 
this one is unique in addressing the information 
and assessment needs of teacher recruitment 
programs. We recommend starting at the 
beginning where we provide an overview of 
the sections comprising the handbook and 
important tips for getting started. Then, simply 
proceed step-by-step, section-by-section until 
you have acquired the basic tools and know-how 
for evaluating your program. Good luck! 

A final note: In order to evaluate the 
handbooks effectiveness and usefulness, we urge 
readers to complete, tear out, and return the 
Reader Response Form provided at the back. 
Please let us know whether the handbook meets 
your needs, what features are most/least helpful, 
and what, if anything, is lacking, so we can 
make revisions in the future. We also invite 
your inquiries or comments by telephone 
(617) 489-6000 or e-mail: rnt@rnt.org. 
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Overview of the Handbook 



I n designing and conducting an evaluation, 
you’ll find that each step builds on the prior 
ones. For example, if you don’t understand 
why you are evaluating, your guiding research 
questions won’t make sense. Similarly, if you 
haven’t clearly identified logical research 
questions based on the purpose of the 
evaluation, there’s a good chance you won’t 
select the most appropriate methods for 
collecting data to address each question. Each 
section of this handbook builds on the previous 
ones and allows you to create and carry out an 
evaluation design for your teacher recruitment 
program. An overview of the sections 
comprising this handbook is illustrated below. 



We suggest that you follow each of these steps in 
logical order to maximize your understanding of 
evaluation as well as the quality of your 
evaluation design and its results. 

In addition to core sections on designing 
and conducting an evaluation, seven mini-cases, 
a glossary of common research and evaluation 
terms, and additional tips from practitioners are 
also provided. For those of you with a desire to 
learn more about evaluation beyond the scope 
of this handbook, a final section provides 
supplemental references to other textbooks, 
handbooks, organizations and resources. 



STEP 8 



Reporting Your Evaluation Results 
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% Tips For Getting Started 



Y ou should now have an understanding of 
the purpose of this handbook and an 
overview of its contents. Before you read 
any further, please review and follow four basic 
tips for getting started in using this handbook. 



1 . Bring together your "evaluation team.” 




While you may be the one who will conduct the 
evaluation of your teacher recruitment program, 
it may be useful to bring together others 
involved in the program (e.g., program staff and 
a member of the advisory board) who can lend 
insight and assist in carrying out the evaluation 
steps contained within this handbook. This 
will not only be a learning process for you, it 
will also help other program staff gain an 
understanding of evaluation — how to conduct 
one, how to use resulting information to better 
conceptualize the program, as well as ways to 
improve it. In identifying members of the 
evaluation team, make sure they know what it 
will entail in terms of time demands, timeline, 
their role, etc. and secure a commitment from 
all members before starting the process. 



2. Review the chapters of the handbook 
with the team. 



Before attempting to design and carry out an 
evaluation, you and other evaluation team 
members should spend some time reviewing the 
steps outlined in each chapter of this handbook 
as well as the accompanying worksheets. By 
doing so, you will have a clearer understanding 
about evaluation’s purpose and how it relates to 
your needs. Also, working through this 
handbook will take some dedicated time (and 
energy). We recommend that you build a 
realistic schedule for conducting the various 
parts of your evaluation. Even if you have 
developed an excellent evaluation plan, you may 
be disappointed with the results if you do not 
leave yourself sufficient time to execute the plan. 



3. Create a realistic timeline for conducting 
the evaluation. 

It is a good idea to have an understanding of the 
most appropriate and effective time for 
conducting the program evaluation. Part of this 
planning depends on why you need to evaluate 
and how you will use the information. For 
example, if you are seeking evaluation 
information that you can incorporate into a 
grant proposal due in six months, you will have 
to determine not only the type of information 
you require from the evaluation, but the scope 
of the evaluation itself. In other words, you 
need to make sure that you can design and 
implement an evaluation plan that collects the 
right type of information within that period of 
time. 



4. Procure all "go-ahead” and permissions. 

If you believe that you need approval from 
specific people or organizations before starting 
the evaluation (e.g., from school administrators, 
school board, funding agency, etc.), it is wise to 
secure their permission now so as not to get 
bogged down with “red tape” later in the 
process. 
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Eight Steps to Successful Program Evaluation 




evaluating, there exist 

I s " ' ' s " ' ' 



T- , v . 

different types and 

. 

levels of evaluation, 

"V;.: -V i f. ! : r" ^ T 

depending on its 
purpose. Examples 
include: program, 
project and planning 
evaluations, formative 
and summative 
evaluations, and 
implementation and 
progress evaluations. 
Program evaluation 
looks at educational 
activities provided on a 
continuing basis and 
project evafua tion looks 
at educational activities 
provided for a defined 
period of time (Sanders, 
1994). See Glossary for 
more information on 
different evaluation 
types. 
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Understanding 
the Purpose 
of Program 
Evaluation 



What is This Thing Called Evaluation? 

The term evaluation takes on many meanings 
depending on the context in which it is used. 

No one definition describes adequately the 
process of evaluation. A commonly accepted 
definition of evaluation, however, is: 

“The formal examination of the quality, 
effectiveness , or value of a program, 
product, project, process, objective, or 
curriculum.” (Worthen &; Sanders, 1988; 
see Resources section) 

No matter how evaluation is defined, an 
important first step in using this handbook is to 
put aside any notions you might have about 
evaluation being “bad.” There is quite a difference 
between the traditional image of being evaluated 
and the actual process of conducting evaluation for 
self- reflection and program improvement. This 
handbook provides you with the basic methods for 
designing and implementing an evaluation plan. 
Using the evaluation information you have 
collected, you will not only have the capacity to 
identify and measure indicators of program 
success, you will also be better equipped to 
uncover programmatic needs and gaps and to 
implement strategies for addressing them. In 
essence, you and your staff will become 
empowered to incorporate evaluation skills into all 
aspects of your program environment and to 
become reflective self-evaluators. 

Whether you know it or not (and whether 
you like it or not!) you are already evaluators of 
your teacher recruitment program. You are 
evaluators because you are in the process of 
evaluating when you ask the following types of 
everyday questions about your program: 

• What is the issue we are trying to address? 



• What are we doing to address this issue? 

• Are we doing what we said we. would do? 

• Are we doing these things well? 

• How should we improve what we do? 

• What difference have we made? 

More specifically, you may wonder: 

• Is our teacher recruitment program really 
helping to prepare participants for a career 
in teaching? 

• Are we reaching potential program 
participants? 

• Should we consider eliminating or changing 
certain components of the program? 

• How can we reach more students/ 
participants? 

• How many of our program participants 
succeeds the program? 

• What happens to program participants when 
they complete the program? 

• How can we be more efficient in operating 
this program? 

• Can this program serve as a model ? What 
would other people need to know about it 
in order to adapt it? 

These questions focus on a teacher 
recruitment program that is in place, where 
resources have been dedicated and program 
activities have been designed and implemented. 
You are now at a point in your programs life where 
you 11 want to examine the processes of your 
program (e.g., the activities and specific 
components) and assess impact (or outcomes). 
Even if your program is in the early stages of 
implementation, it is never too early to assess the 
effectiveness of activities and track outcomes, 
because it is likely that at some point, someone or 
some group will ask, “How did your program 
make a difference in the lives of its participants?” 
Before we delve into how to evaluate, you 
need to determine exactly what you intend to 
evaluate. 
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Worksheet #1A 



Describing and Prioritizing 
What is to be Evaluated 



Describing the object of evaluation — in your 
case, your teacher recruitment program — is the 
critical first step to any evaluation design. In 
working so closely with a program, it is easy to 
lose perspective on exactly what you are doing 
and why. Furthermore, different people 
involved with the program (e.g., program 
director, program staff, participants) may view it 
differently depending on their own unique 
perspectives. Before delving into the evaluation 
design, it is important to understand the 
program from these multiple perspectives. In 
other words, this first step allows you to pause 
and simply describe what it is you will be 
evaluating. 

There are five simple elements to the 
program description: 




• WHO: Who’s involved? Who are the key 
decision makers and leaders, the funders, 
the personnel implementing the 
program, and those served by the 
program? Who are the advocates and 
adversaries, partner organizations, or 
other interested groups? 

• WHY: Why does this program exist? 
What are the goals and objectives of the 
program? What needs are being 
addressed? 



• WHAT: Is there a model or description 
of the program? What and how many 
separate components are included? 

What kinds of activities are included? 
What services are provided? What are 
the potential benefits and opportunities? 
Generally, what appear to be the biggest 
issues or problems in the eyes of 
program personnel and other 
stakeholders? 



* WHEN: How long has the program 
been around? What does its history look 
like and what kind of future is it 
anticipating? How long does it have to 
accomplish short- and long-range goals? 

* WHERE: What is the setting of the 
program? Where does it fit into the 
larger organization? 
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Program Evaluation Design 

Make a copy of Worksheet #1A for each member 
of the evaluation design team. Have each person 
individually describe the program to be 
evaluated from his or her perspective. Once 
everyone has completed the worksheet, review i 
the comments as a group. Use this time to 
discuss, brainstorm, and build consensus on 
what is to be evaluated. 



Again, Why Evaluate? 

Based on the information you’ve collected for 
Worksheet #1A, you now know what you are 
evaluating. Before you put the time and energy 
into designing an evaluation plan for your 
program, however, you need to ask: “ Why 
evaluate this program?” 

Whether the program is new and still under 
development or one that has been around for 
years, depending on your information and 
decision-making needs, there are several reasons 
for conducting an evaluation: 



LOOK AT... 


IN ORDER TO... 


A. Goals or Needs 
Identification 


Determine whether goals 
and needs have been met 


B. Program Design 


Determine the adequacy 
of the program design 
that was used; compare 
the design to alternatives 


C. Process or 
Implementation 


Determine whether it was 
implemented as planned; 
describe problems that 
developed; examine what 
happened vs. what was 
expected 


D. Results/Outcomes/ 
Impact 


Assess the quality of 
outcomes; determine the 
outcomes achieved; 
determine if it was 
worthwhile 


E. Decisions about 
Continuing 


Determine if it was worth 
the resources it consumed 
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Worksheet #1 A: Describing What is to be Evaluated 

Describing the Program (Defining What You'll Evaluate) 
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Measured Steps 



Where does it exist? (as a single entity and in relation to the whole organization?) 













While these are all important reasons for 
engaging in evaluation, this handbook will 
focus on parts C & D — processes and 
outcomes. Why focus on the evaluation of 
processes/implementation and the 
measurement of impact? As mentioned 
earlier, if you are using this handbook, you are 
probably at a point where your teacher 
recruitment program is in place, where 
resources have been dedicated to running the 
program and program activities have been 
designed and implemented. It is inevitable 
that someone or some group with a stake in 
your program (e.g., a funder, a recipient, a 
supporter, etc.) will want to know if all the 
time and resources dedicated to the program 
really make a difference. In other words, you 
are or will be accountable for talking about 
the impact of your program; and to measure 
impact you must be familiar with the 
processes, including the activities and other 
components of your program. 

Accountability is an important reason for 
It examining the processes and outcomes of your 
program, but it is not the only reason. An 
even more important reason for examining 
processes and measuring outcomes is to help 
you to make decisions about program 
improvement. The ongoing process of 
assessing implementation and measuring 
outcomes will provide you with important 
feedback about program processes that will 
allow you to identify gaps and make “mid- 
course” strategic changes to your program. In 
so doing, you will become more effective and 
efficient in carrying out program objectives 
and in meeting the needs of participants. 
Other benefits may include: promoting the 
program to potential participants, enhancing 
the program’s public image, and retaining and 
increasing funding and other types of support. 

Identifying Processes and Impact 

Identifying program processes and impact (or 
outcomes) will literally guide each subsequent 
step of your evaluation, from identifying 
research questions to designing the 
appropriate data collection instruments and 
making recommendations based on findings. 




CHALLENGE Measuring the long-term 
impact of your program is perhaps the most 
elusive evaluation issue and critical challenge 
you will face in the evaluation process. The 
ultimate goal for any teacher recruitment 
program is that a good number of program 
participants go on to become successful 
teachers. Yet this is a most difficult goal to 
measure, as it takes years for a high school, 
middle school, or even a college student to 
traverse the pipeline into a teaching position. 
What’s more, even if program participants 
could be tracked over time, determining the 
direct impact of the teacher recruitment 
program on the participant is yet another 
challenge. 

SOLUTION Identify interim goals that will 
indicate progress toward teaching. For example, 
you can measure participants’ progress in 
school, retention in your program from year 
to year, college acceptance and/or entrance 
rates, attitudes toward teaching and career 
goals over time, etc. All of these measures, in 
the aggregate and tracked over time, provide a 
solid profile from which to draw conclusions 
about program impact. 



Program processes relate to the specific 
aspects of program components and the 
implementation of those components. In 
assessing processes, you might look at: 

• Attendance at program activities 

• Participant satisfaction with program 
activities/support structures/etc. 

• Retention or participation in the program 

Program impact (or outcomes) relates to 
what happens to the participants as a result of 
being in the program and being exposed to the 
various program components. Program impact 
may relate to changes in knowledge, skills, 
attitudes, values or behaviors. In assessing 
program impact, you might look at: 

• Attitudes about teaching as a career 

• Grades or academic performance of 
program participants 



ERIC 









=^?jcruiting New Teachers, Inc. 





11 



• Participant confidence and self-esteem 

• College matriculation rates of program 
participants 

• Numbers of participants who eventually 
enter the teaching field 

While the ultimate outcome of your 
program is its impact in terms of the number of 
participants entering careers in teaching, you 
may have to measure interim (short-term) 
outcomes over a period of time (see Challenge 
& Solution box on the previous page) to collect 
the kinds of data that will help you to 
hypothesize about the long-term impact of your 
program. 

However, as your program matures and as 
participants complete the program and continue 
on with their educational and professional plans, 
you (and the various other stakeholders) may 
want to know what happens to participants after 
they leave the program. In this case, you should 
consider alumni or follow-up studies. 

Follow-up studies involve tracking and 
collecting information from your participants 
over time. For example, many high schools and 
college engage in follow-up studies of their 
graduates that entail surveying them one, two, 



five, ten or more years after graduation. The 
focus of the follow-up surveys is not only on the 
current endeavors of alumni but also how the 
program (or school, educational program, etc.) 
impacted what they are currently doing and 
what they plan to do. Schools then use this 
information for such things as advocacy 
building, public relations, program 
improvement, fund-raising and so forth. 



Worksheet #1B 



Identifying Program Processes 
and Impacts 



Make a copy of Worksheet #18 for each member of the 
evaluation design team. Individually or as a group list 
the process and impact outcomes for your evaluation 
and then have each person individually rank each 
outcome in order of importance. Use this ranking as a 
point of discussion for deciding priorities for 
evaluating. You may not be able to do everything now, 
but ranking will help you to decide where best to place 
your focus and resources in order to meet your 
immediate evaluation needs. 
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Identifying Audiences 
for the Evaluation 



Who’s Interested in This Anyway? 

So, now you want to know who else could 
possibly be interested in an evaluation of your 
program. Well, youd be surprised by the 
number and types of stakeholders out there! 

A stakeholder is any person or group of 
persons who can impact or be impacted by your 
program, directly or indirectly. Who are these 
people? They include audiences who: 

• make decisions based on the evaluation 
findings; 

• are involved in planning or modifying 
the program being evaluated; 

• might be affected by the evaluation 
results; 

• have legal rights to know about the 
evaluation; 

• sponsor, commission or pay for the 
evaluation; 

• advocate or criticize the program being 
evaluated; 

• provide information for the evaluation; 

• run the program being evaluated. 

Roles of persons or groups who might be 
audiences for your teacher program include: 

• principals/superintendents 

• foundations 

• boards of education 

• legislators 

• taxpayers 

• funding agencies 

• teachers 

• community groups 

• parents 

• adversaries 




• friends/advocates 

• unions 

• news media 

• project directors/staff 

• students/program recipients 

• higher education administrators/faculty 

Although all of these audiences may not be 
involved directly with the evaluation, it is 
important for the evaluation design team to 
identify (within reason) all audiences or 
stakeholders who could have an interest in the 
evaluation and its results. 

There are many ways to identify these 
multiple interests. For example, you could: 

• have conversations with key program 
staff; 

• consult the program advisory board; 

• have conversations with groups outside 
the program; 

• make observations of key program 
activities and players; 

• review documents that define audiences; 

• consult program funders. 



Worksheet #2 



Identifying and Analyzing 
Audiences/Stakeholdersn y 

Make a copy of Worksheet #2 for each member 
of the evaluation design team. Individually or as 
a group list the audiences/stakeholders for your 
evaluation and describe the particular values, 
interests, or expectations these individuals may 
hold which are important to consider as you 
undertake your evaluation. 
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Worksheet #2: Identifying and Analyzing Audiences/Stakeholders 
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Planning for Evaluation 
Resources and Costs 



How Much is This Going to Cost? 

People are more likely to dread the cost of 
conducting an evaluation than to recognize the 
cost of NOT evaluating their programs or to 
understand the potential cost-benefit of 
conducting evaluation. Evaluations that focus 
on program improvement , for example, can result 
in cost savings and quality gains by indicating 
ways to improve program processes without 
increasing program costs. Evaluations that focus 
on program outcomes can be used to promote the 
program and rally support from a range of 
stakeholders, which could result in continued 
funding and other forms of support. 

It is difficult to predict the a^amount 
it will cost you to design and conduct an 
evaluation for your teacher recruitment 
program. This is because the cost of conducting 
evaluation depends on a combination of factors, 
including the size of your program, the scope 
of the evaluation and the methodologies used, 
and decisions about who will conduct the 
evaluation. What is provided here are some 
issues for you to consider when analyzing the 
resources and capabilities that can be committed 
to your program’s evaluation. 



What Makes Up the Cost of Conducting 
an Evaluation? 

Your evaluation budget depends on the scope of 
the evaluation and the type of information you 
wish to collect. The costs associated with 
conducting an evaluation often revolve around 
the following resources: 

• Personnel — includes the people who will 
conduct the evaluation and support 
evaluation activities; 

• Technology Support — includes 
equipment (e.g., computers, Scantron 
machine and forms), software, training, 
maintenance; 

• Material — includes supplies, 
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photocopying, production of evaluation 
reports, etc.; 

• Communication — telephone charges for 
phone interviews and survey follow-up 
and postage for mailing questionnaires; 

• Travel — includes mileage or airfare for 
traveling to program sites; 

• Meeting Expenses — includes meeting 
facilities, refreshments, etc. for 
interviews or focus groups; presentation 
of evaluation results, etc. 

Obviously, the amount expended on any of 
these resources depends on these factors: 

• The size of the program and the scope of 
the evaluation; 



• The types of evaluation methodologies 
used (e.g., surveys, interviews, focus 
groups, case studies); 

• Decisions regarding who will conduct 
the evaluation (internal vs. external); 

• The types and amount of resources that 
may be acquired as “in-kind” support 
from funders, staff, or other sources. 
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CHALLENGE A frequently cited evaluation 
challenge related to resources is time — as in, 
not enough of it! In general, teacher 
recruitment programs do not have large staffs, 
which makes it difficult to find time to plan 
for and implement program activities, let alone 
collect data for evaluation! 

SOLUTION Viewing evaluation as an integral 
part of your program and incorporating 
evaluation activities into program operations can 
be an efficient and effective way of collecting 
data. For example, distribute participant 
satisfaction questionnaires at key points 
throughout the year and use information 
gathered from these forms to design an end-of- 
year (or end-of-program) questionnaire. In lieu 
of a questionnaire, you might consider reviewing 
participant journals, which are not only a very 
valuable program activity and learning tool, but 
also a potentially excellent source of evaluation 
information. 
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Identifying Other Low Cost/No Cost 
Resources 

Examples of other resources that may be 
available to you at little or no cost include: 

• The availability of existing data. The 
more primary data that need to be 
collected, the more costly the evaluation. 
Any existing files, participant records, 
previous evaluation information, etc. that 
could inform your evaluation should be 
identified and considered. (Also see the 
Challenge & Solution box on the previous 
page.) 

• The availability of existing data 
collection instruments. Existing tests, 
questionnaires, interview protocols, and 
the like are resources that will cost you 
little or no money, but should be used only 
/jThey fit the scope of the evaluation. 

• The availability of existing support 
services. Depending on where your 
program operates, you may have several 
support services (such as computer 
programming and database support, 
duplication and printing services, 

Scantron machines, etc.) available to you 
at little or no cost. These services should 
be considered in the implementation of 
the evaluation design. 

• The availability of evaluation staff. 

You may be able to make use of qualified 
personnel on-site for carrying out your 
evaluation activities. Examples include 
using teachers to collect data or secretaries 
to type and enter data into databases and 
spreadsheets. In addition, graduate 
students seeking internships or data 
collection opportunities for dissertations 
can serve as evaluation staff at little or no 
cost. Graduate students in the social 
sciences are often skilled in data collection 
instrument development (such as surveys 
or interview protocols), conducting 
interviews and focus groups, data entry 
and statistical analysis, and data 
interpretation. Contacting a college or 
university in your area is a good place to 
start. 



A Final Tip... What You Should Know 
about Hiring an Outside Evaluator 

While this handbook assumes you will be 
conducting the evaluation internally , it is 
nonetheless important to have a general idea 
about identifying and hiring an external 
evaluator in case you choose that route at some 
point in the future. A reliable and expert 
external evaluator is a good choice if you are 
interested in knowing about program outcomes 
or impact (e.g., “What difference has this 
program made?” “Should it be continued?”). An 
expert from the outside provides an element of 
credibility to the evaluation and reduces the 
threat of bias or conflict of interest that comes 
with evaluating ones own program. Using an 
external evaluator also adds an outside 
perspective and allows you to view the program 
through a different lens and see things that may 
otherwise be missed by someone working closely 
with the program on a daily basis. 

External evaluators who possess formal 
academic preparation , evaluation experience , and 
a strong track record for conducting evaluation 
also bring with them the technical and 
professional expertise necessary to ensure the 
collection of valid and reliable data and the 
effective communication of evaluation results. 
Note: Be sure to check the references of any 
external evaluator you are considering. 

An external evaluator will undoubtedly 
be more expensive because of the associated 
professional costs and the usual added 
communication and travel costs. It is up to 
you to weigh the benefits and costs of using 
external evaluators based on your program needs 
and resources. 




If you do decide to 

use an external 

■■ . 

evaluatoiVlbe sure to 
communicate your 
budgetary limits and 
concerns, so that 
• together you can 
determine a realistic 
plan in relation to 
your resources. 

Also, ask your 
evaluator to present 
a “menu” of 
evaluation options 
within your budget, 
so that you can pick 
and choose the plan 
most suited to your 
decision-making 
needs. 



Worksheet #3 




Identifying Evaluation Resources 








Make a copy of this worksheet for each member of the 
evaluation design team. As a group, use Worksheet #3 - 
to identify possible resources for carrying out your 
evaluation, including funding, personnel, support 
services, and in-kind contributions. 
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Worksheet #3: Identifying Evaluation Resources 
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Identifying 
Evaluation Questions 



The Question Is: What Questions 
Need to Be Asked? 

The evaluation questions are the guide to good 
evaluation design and activities. Without sound 
evaluation questions, the evaluation will lack 
focus and evaluation results will be disjointed 
and ambiguous. Furthermore, along the way 
you will have wasted time and resources, 
misdirected future efforts, and lost credibility 
with your audiences and/or stakeholders. 



Identifying the questions and issues that will 
guide your study takes time, reflection, and the 
involvement of key stakeholders. In identifying 
evaluation questions, it is best to brainstorm a 
range of questions rather than to place limits on 
the number and type of questions to be included. 
Questions included in your evaluation design 
should be guided by the processes and impacts you 
want to measure. The following matrix provides 
you with basic evaluation questions based on some 
examples of program processes and impacts. 



Processes & Impacts 


Typical Evaluation Questions 


Retention in the program (Process) 


What are the characteristics of participants? 
Who leaves the program and why? j 

How has retention changed over time? Why? 


Quality and relevance of program 
activities/components (Process) 


What activities are most/least effective and why? 
What mid-course directional changes have 
been taken and why? What was the result? 
Are there sufficient resources to implement 
the activities? 


Academic achievement of program participants 

(Impact) 


What are the academic outcomes of this 
program in terms of participants’ grades? 

How prepared are participants to attend 
college? 

How many intend to go to college and how 
has that changed over time? 


Changes in participants’ attitudes, confidence 
and self-esteem (Impact) 


How has the program helped participants to 
identify goals and ways of reaching the goals? 

How has the program provided support to 
participants in reaching their goals? 

How has participants’ motivation and 
confidence to attain goals changed over time? 


Participants’ career plans (Impact) 


What percent of participants plan to teach 
upon entering and leaving the program and 
how has that percent changed over time? 

How have participants’ attitudes about 
teaching changed over time? 


Participants entering the field of teaching 

(Impact) 


What is the number of participants that enter/ 
graduate from a teacher preparation program? 

What is the number of participants who enter 
the classroom to teach? How does that 
number change over time? 
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Once you have drafted a thorough list of 
evaluation questions for each of the outcomes 
you want to measure, use a set of criteria to 
determine which evaluation questions should be 
investigated in more detail and serve as the 
foundation of your evaluation design. 

A well-known set of criteria (Cronbach, 
1982) for establishing a final set of evaluation 
questions includes: 

1. Who would use this information? 

Who wants to know? Would an 
important audience suffer if this 
question were deleted? 

2. Would an answer to the question 
provide information that is not 
currently available? 

3. Would the answer to this question 
yield important information about the 
program (i.e., “need to know” versus 
“nice to know”)? 

4. Is this question of passing interest to 
someone, or does it focus on critical 
dimensions of continued interest? 

5. Would the scope of the evaluation be 
seriously limited if this question were 
dropped? 



6. Would the answer to this question 
have an impact on the course of 
events? 

7. Is it feasible to answer this question 
given available financial and human 
resources, time, methods, and 
technology? 



Worksheets #4A, 4B, 4C 



Identifying Evaluation Questions 



Make a copy of these worksheets for each member of 
the evaluation design team. As a group, use Worksheet 
#4A to draft your original set of evaluation questions 
using the matrix on the previous page, as well as the 
information you gathered in Worksheet #1 B: Identifying 
Program Processes and Impact as guides. Then use the 
matrix in Worksheet #4B to rank these questions based 
on the set of criteria provided by Cronbach (see above) 
and add your own criteria, if necessary. When you have 
completed ranking your questions, use Worksheet #4C 
to list a final set of evaluation questions that will guide 
the study. 



BEST COPY AVAILABLE 
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Worksheet #4A: Brainstorming and Drafting Questions 

(Deciding What You Want to Know) 




Worksheet #4B: Ranking Questions Based on Criteria 
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Selecting Methods 
for Data Collection/ 
Sample Surveys 



Identifying Indicators and Collecting 
Information to Measure Outcomes 

Indicators provide specific information that 
allows you to gauge a programs attainment of 
desired outcomes. They describe observable, 
measurable characteristics or changes that 
together represent achievement of an outcome. 
For example, a program with a desired outcome 
of preparing participants to pursue a career 
in teaching could define “prepared” as 
maintaining the grades necessary to enter 
college; demonstrating knowledge about the 
process of getting into college (e.g., selection, 



application, financial aid, class scheduling, etc.); 
demonstrating a desire/intention to teach; 
demonstrating a positive attitude toward 
learning and education, etc. The number 
and percent of program participants who 
demonstrate these behaviors then is an indicator 
of how well the program is doing with respect 
to the outcome of being “prepared” to teach. 

Here are some examples of indicators based 
on some process and impact research questions: 



Desired Outcome 


Research Questions 


Some Indicators 


Program components 


(Process Research Question) 


9 Number of participants 


adequately meet the goals 


What activities were most/ 


attending 


of the program. 


least effective? 


• High vs. low ratings on & 

activity follow-up | 

questionnaire 'J 

• Examples of information * 

usage |j 


Participants are prepared to 


(Impact Research Question) 


1 

| 

• Evidence of intentions to | 


pursue a teaching career. 


Are participants well prepared 
to eventually become 
teachers? 


teach 1 

• Number of participants \ 

that graduate from high ( 

school and enter college j 

• Changes in participants’ 
knowledge about what it 
takes to be a teacher 

• Changes in participants’ 
attitudes about teaching 
and learning 
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Collecting Data on the Indicators 

As an evaluator, you have several options for 
choosing the methods you will use to collect 
information based on the indicators you have 
selected for each outcome. The challenge in 
data collection comes with selecting data 
collection methods that allow you to maximize 
the information you receive while minimizing 
the time it takes to collect the information. This 
includes the time it takes to develop the data 
collection instrument, the time it takes for the 
participants/respondents to provide the 
information, and the time it takes to get all of 
the data together for analysis. Typical evaluation 
data collection methods include: written or 
phone surveys and questionnaires, one-on-one 



or group interviews (focus groups), in-depth 
case studies and observations, and tests. While 
the indicators you identify and the ultimate 
outcomes you hope to examine will drive the 
data collection methods you use, other factors 
will also play a role. For example, in choosing 
an evaluation method for data collection, you 
should consider: the cost and resources for using 
it (e.g., a case study can be costly in terms of the 
evaluators time); the precision and reliability of 
the survey instrument; efficiency; and 
acceptability to various audiences. Some 
examples of data collection methods based on 
the type of information you want to collect 
include the following: 




Information 
Desired -» 


Quantity & Quality 
of Program 
Activities and 
Processes 


Knowledge & Skills 


Behaviors 


Attitudes/Values 


Evaluation 


Participant reports 


Tests or other 


Records 


Participant 


Methods 


of satisfaction/ 


assessments 


(attendance in 


changes in 


and Tools 


adequacy/ 


(pre/post) 


school, attendance 


attitudes/values 




relevance/use 




in program) 


(survey/ rating 




through surveys/ 


Grades 




scales, focus 




rating scales, 




Changes in 


group, interviews) 




interviews, and 


Participant reports 


participants’ 






focus groups 


of knowledge 


behaviors 


Others’ report of 






gained (surveys, 




changed attitudes 




An analysis of 


interviews) 


Others’ report of 






existing records 




behavior (e.g., 


Observations 




. (attendance data, 




parent or teacher 






retention rates, 




survey/interview) 






number of 










participants 




Observations 






attending, number 










of activities, 










number of hours 










of service) 
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CHALLENGE There are many ways that you 
could be blocked from collecting essential 
information. For example, authorities (e.g., 
school administrators) might not allow 
certain information (student attendance, 
parent information) to be collected. 
Participants who mistrust you and what you 
are collecting could give misleading or false 
information if they think it will be used 
against them. Or respondents could 
misunderstand directions and consequently 
respond inappropriately. 

SOLUTION While there is no guaranteed way to 
avoid such problems with data collection, 
there are certain things you can do along the 
way to head-off such problems. With respect 
to data access, make sure that your data 
collection methods are approved by the proper 
authorities. In schools, this may be the school 
principal, the school board, the teachers, and / 
or students’ parents. Obtaining permission and 
cooperation requires that you clearly 
communicate the purpose of the evaluation, 
how information will be used, and how the 
evaluation will serve the participants’ best 
interests. To gain the trust and cooperation 
of participants and other respondents, stress 
the importance of the study and how the 
information they provide will help to improve 
the program. Always indicate that their names 
or other private information will not be shared 
with others and that you will only report on 
“group” responses. In other words, their name, 
and individual responses will be kept confidential. 

Note: Before proceeding, you may want to refer to the 
Family Educational Rights and Privacy Act of 1974 , 
commonly called the Buckley Amendment, which spells 
out students' rights with respect to their educational 
records. 
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A Quick Overview of Some Common 
Evaluation Methods 

As the matrix on the previous page shows, there 
are many useful methods of information collection 
an evaluator could use. An evaluation method 
should be used only after careful thought has been 
given to the information required. If you are 
limited in your evaluation resources and 
expertise, it is best to stick to the basic or “core” 
methods. For this reason, the focus of this 



handbook is on Survey Questionnaires, 
Interviews, and Observations. 

Survey Questionnaires The development of the 
questionnaire (including the format and 
questions) is the most critical aspect of 
conducting survey research. Unless the 
questionnaire undergoes careful development 
and input before it goes to the respondent, it 
may yield little usable information. 

In developing your survey questionnaire, you 
need to focus on: (1) asking the right questions; 
(2) question construction; and (3) format. 

1. Asking the Right Questions 

The key to a successful questionnaire is in the 
art of asking the right questions. No matter what 
questions you decide to ask, the following hints 
will aid you in designing questions that are most 
relevant to demonstrating impact and are 
tailored to meet the characteristics of program 
participants and other respondents. 

• State questions precisely Use language 
that respondents will understand; 
eliminate extra words; keep the 
questions simple, clear and direct. 

• Avoid attitude questions that refer to 
the past rather than the present. Try not 
to ask questions about attitudes or 
feelings that involve recall of the distant 
past because respondents may not 
remember that far back and could 
provide inaccurate information. 

• Avoid questions that could be 
interpreted in more than one way. 

• Ask questions that are relevant only to 
information you need to answer your 
research questions. In other words, don’t 
overburden your respondents with 
nonessential questions. 

• Avoid questions to which you think 
ALL respondents will respond alike. 

Ask questions that will help you to 
distinguish among respondents. 

• Express only one complete thought 
per question. For example, don’t ask, 

“Did you like it and did it improve your 
skills?” in one statement. 
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Keep the sequence and grouping of 
questions logical. Don't “jump around” 
in the sequence of questions; allow 
respondents to maintain a logical train 
of thought. 

• Avoid questions where you think the 
respondent may have to “guess” at a 
response. Don't ask questions that 
assume too much knowledge on the part 
of the respondent. For example, don’t 
ask students to give family income and 
don’t ask parents to indicate what they 
think their child feels about something. 

’ Avoid technical terms, jargon, slang, 
or words with double meanings. 

’ Make sure each question can be 
answered within a reasonable time 
period. 

’ Avoid using open-ended questions on 

a survey if you can get the same type of 
information from closed-end questions. 
(See Glossary for definitions of open- 
ended and closed-ended questions.) 



2. Question Construction 

In the construction of questionnaire terms, there 
are basically two types of questions: open-ended 
and closed-ended questions. 

You can ask open-ended questions that ask 
the respondents are to provide their own 
answers to the question. For example, 
respondents might be asked, “What was the 
most useful activity of the program for you and 
why?” and be provided with a space to write in 
their own answers. 

In closed-ended questions, respondents are 
asked to select their answer from a list provided 
by the survey developer. Closed-ended 
questions are used more frequently with surveys 
because they provide a greater consistency of 
responses and are more easily processed. 

Whereas open-ended questions will require the 
evaluators to read through each response and 
develop a coding plan based on responses, 
closed-ended responses can often be entered into 
a computing program for analysis directly from 
the questionnaire. 

A drawback to closed-ended questions lies 
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in the structuring of responses. Careful 
attention to the choice of responses is necessary 
to ensure that all relevant answers are covered. 
Earl Babbie (see Resources ), a well-known survey 
researcher, offers two guidelines that should be 
followed in the construction of closed-ended 
questions: 

• The response categories provided should 
be exhaustive , that is, they should 
provide all possible responses that might 
be expected. To allow for responses that 
may have been overlooked, researchers 
often include a category labeled “Other 
(please specify).” 

• The answer categories must be mutually 
exclusive , that is, the respondents should 
not feel required to select more than 
one. You can ensure mutually exclusive 
answers by carefully considering each 
combination of responses and asking 
whether a person could reasonably give 
both. 

In some cases, you may want to have more than 
one response (e.g., “Check all factors that 
influenced your decision to teach”), in which 
case you would specifically ask the respondents 
to “check all that apply.” If, however, you are 
looking for the “most important” or “most 
influential” factor, then you must ensure that 
your responses are mutually exclusive. 

Another form of closed-ended questions is 
contingency questions. Often in survey 
research, you will be interested in the responses 
of certain sub-groups of the respondents. For 
example, you might ask respondents if they 
participated in a specific activity and then ask 
those who did what they thought about it. You 
might be interested in the influence of certain 
aspects of the program on males, or you might 
ask respondents who are currently teaching (i.e., 
alumni of the program) what specific aspects of 
the program impacted their decision to teach. 
These questions are referred to as contingency 
questions because they are contingent upon the 
first question - group membership. In other 
words, only those who answer “yes” to the first 
question (e.g., from the examples above: 
participants of a specific activity, males, those 
entering the teaching field) will be asked to 
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continue with the follow-up question. An 
example of a contingency question follows: 

Did you participate in the classroom internship 
while a participant in the program? 

□ No 

□ Yes — if Yes, what was the most effective 
feature of the internship activity? 

□ Practical experience in the classroom 

□ Earned school credit 

□ Learning from teachers 

□ Other (specify) 

Another form of closed-ended questions is the 
Likert scale, which is commonly used if you are 
interested in determining the extent to which 
respondents hold a particular attitude or 
perspective. If you are able to summarize the 
attitude in a fairly brief statement, you will often 
present that statement and ask respondents 
whether they agree or disagree with the 
statement. Include four or five possible 
responses: “strongly agree,” “agree,” “disagree,” 
“strongly disagree,” and sometimes, “don’t 
know.” (Excluding the “don’t know” response 
forces the respondent to make a choice.) You 
could also use “strongly approve,” “approve,” 
and so forth. 

In addition to asking the right questions, 
careful attention to the following questions is 
essential in survey development: 

3. Survey Format 

• Is the questionnaire attractive and 
interesting? 

• Does the questionnaire start off with 
easy, non-threatening questions? 

• Are questions with similar content 
grouped accordingly? 

• Are all the major issues covered 
thoroughly? 

• Is the questionnaire easy to answer? 

• Is the time required to respond 
reasonable? 

• Is the questionnaire introduced with an 
explanation of purpose, assurance of 
confidentiality, etc.? 



• Is the respondent told clearly how to 
record his/her responses? 

• Are instructions for return due date and 
procedures included? 

• Have content experts or others involved 
with the program reviewed the 
questionnaire and given input? 

• If this is a telephone survey, have phone 
surveyors been trained in the correct use 
of the questionnaire? 

Examples of survey questions and basic 
questionnaires are provided later in this section. 
A bibliography on the design, construction, and 
use of surveys can be found in the Resources 
section. 

Interviews An advantage of collecting data 
through face-to-face or telephone interviews 
rather than through a questionnaire is that 
interviews allow for probing and clarification as 
well as the generation of broad and deep data 
about the topic of interest. Face-to-face 
interviews also allow you to build rapport with 
your participants/audience. A downside to 
interviews is that they take more resources (e.g., 
time and staff) to conduct and often require 
training and practice sessions. In addition, 
open-ended interviews, which generate a lot of 
qualitative data, may be more difficult to 
analyze. 

Interview methods are appropriate when 
you want to increase the breadth of your 
understanding of what you are evaluating. 

Even if you want the interview to be open- 
ended and probing, it is a good idea to go into 
the interview with some guiding interview 
questions around the purpose of your 
evaluation. You will also need to be prepared 
with a “preamble” that describes such things as 
the purpose of the evaluation, the purpose of the 
interview, your credibility and competency to 
conduct the interviews, as well as the 
confidentiality of the interviewees’ name, 
statements, and opinions. Here is a sample 
preamble that addresses these issues: 

Hello, my name is: . Iam 

the [your role] of [your program name]. We are 
currently conducting an evaluation of this program 
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to understand better the participants experiences 
while in the program and how the program has 
helped them in their post-high school activities. Im 
here today to ask you general questions about your 
experiences in this program. I'd like to get your 
perspective about the various components and 
activities of this program, and discover what you 
believe works, what doesn't and why I'll ask for no 
more than 30 minutes of your time and promise 
that, by the end of our 30 minutes, we'll both have 
a better understanding of how this program 
benefits participants and what improvements need 
to be made. Your help will make an important 
contribution to this evaluation study. It's 
important for you to know that your name will not 
be attached directly, or indirectly, to anything you 
tell me. I will compile the data from all the 
interviews and pull out themes and generalities. 
This process does not leave space for discovering 
u who said what" and if at any point you say 
anything you re uncomfortable about, let me know 
and I will take extra care to disguise, or remove, 
your comment. Do you have any questions about 
the conversation we are about to have ? 

Following are some more helpful hints for 
conducting interviews: 

• As with the survey questionnaire, 
provide your respondents with the 
appropriate background information, 
including the purpose of the study, how 
information will be used, how their 
involvement will help, etc. 

• Make sure the language is pitched to 
the level of the respondents. While 
specialists may understand technical 
wording and jargon, the general public 
needs language that is more commonly 
understood. 

• Try to choose words that have the 
same meaning for everyone. 

• Avoid asking long questions with 
multiple components. 

• Do not assume that your respondents 
have first-hand knowledge, especially 
if they are answering about someone 
else, for example, a mother for a child. 

• Ask questions that are not positively 



or negatively charged. In other words, 
don’t lead the respondent; maintain 
neutrality. 

• Avoid asking questions that can be 
answered with t<: yes” or “no.” 

Interviews are used to gather more in- 
depth information. If you are interested 
only in “yes” or “no” responses, then use 
a survey questionnaire. 

• Keep the interview time appropriate. 
Don’t impose on your respondents’ 
time. 

• If you audio tape the interview, MAKE 
SURE your respondent knows and gives 
permission to do so. 

Observation Methods As a program coordinator, 
you are in a position to observe key elements of 
the program and its participants every day. 

Some advantages of observations include: 
generating data about actual behavior, not just 
reports of behavior (as in a survey or interview); 
recognizing problems that may not easily or 
accurately be described by participants; and, 
following action at different points in the 
program. There are, however, some 
disadvantages as well. For example, data can be 
skewed by the observers opinion; data are not 
easily quantifiable; and the presence of the 
observer can alter the participants’ actions and 
behaviors. If you plan to observe overtly, it is a 
good (and ethical!) idea to inform participants 
of your presence and purpose. Plan a statement 
for those you observe which includes: the 
purpose of the evaluation; how long you will be 
observing; and assurances that you are there to 
study the processes of the program, not 
individual performance. 

While evaluation observation can involve 
detailed methods, there are some simple, low- 
cost observational methods you can employ as a 
program coordinator that yield important 
information about your program. Some 
methods for qualitative observation include the 
following: 

• Running Notes: jotting down observations 
about your program and its participants as 
they occur 
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• Log or Diary: taking detailed notes 
about your program on an ongoing basis 

• Notes on Themes: writing notes on a 
particular theme or themes about your 
program and its participants 

• Panels: making periodic observations of the 
same participants over time 

Qualitative observations, along with other 
data collection methods such as surveys or 
interviews, allow for a more complete picture of 
what is happening in the program. In addition, 
reviewing your observations and looking for 
patterns will help to reveal issues or concerns 
that can be explored further through a survey or 
interview. 

In deciding which methods to use, you will 
also need to consider at what points in the 
program you would use each method. For 
example, if you are interested in examining the 
change in students’ attitudes over time, you will 
need to consider the appropriate points (e.g., 



before and after an important intervention, at 
the beginning and end of the program, etc.) at 
which you will administer a pre-post survey or 
pre-post interviews. If you are using 
observations, then you can track changes though 
your observational “stories” on an ongoing basis. 



Worksheet #5 



DevelopmkanlnformationCollection Plan 



■ . . " 



Make a copy of this worksheet for each member of the 
evaluation design team. As a group and based on your 
Research Questions, use Worksheet #5 to identify 
potential indicators, information collection 
approaches, respondents, and a collection schedule. 

A list of some common methods is provided for your 
use in the box following Worksheet #5. 1^:'”' ! 
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Some Information Collection Procedures 




Procedure 


J • • v W w • V S w.v • . s- V, .ywv - y.y.- s w • * w ^ w >v • w.vtvw a 

What It Measures or Records 


Example 


Opinion Survey 


Opinions and attitudes 


Students are asked to rate the 
quality and relevance of program 
components and activities. 


Survey Questionnaire 


Demographic characteristics, self-reported 
variables 


Frequencies of key behaviors and 
attitudes of students are charted 
over the course of the program. 


Interviews, Group 
or Individual 


Person's responses and views 


Program coordinator interviews 
students about program adequacy. 


Observation Checklist 


Particular physical and verbal behaviors and 
actions 

Environment or context in which program operates 


Session observer records how 
frequently participants use a new 
skill learned, how they work as a 
team, how they interact with 
children, etc. 


Case Studies 


The experiences and characteristics of selected 
persons in a program 


A few students from the program 
are visited at school over a period 
of time. Peers, parents, teachers, 
etc. are interviewed. 


Expert Panels, Hearings 


Opinions, interpretations 


A panel of classroom teachers 
reviews program components and 
give interpretations about 
relevance, quality, etc. 


Records Analysis 


Records, files 


Student attendance records are 
analyzed to detect trends before 
and after program. 


Logs 


Own behavior and reactions are recorded 
narratively 


Program participants maintain a 
log of their thoughts about 
teaching, education, goals, the 
program, etc. 


Knowledge Tests 


Knowledge and cognitive skills 


Students are tested on knowledge 
of teaching practice. 
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Give Me Something I Can Use! 

Sample Survey Instruments 

The quality of data collected and the ability to 
analyze and interpret key information about 
your program depend on the precise 
construction of your survey instrument and on 
each survey question. 

Each question must be clear and 
unambiguous to the survey respondent. 
Consideration must be given to the respondents 
ability and willingness to provide accurate 
information. To help facilitate not only a high 
response rate, but a high quality response, it is 
essential that the survey focus on the most 
critical “need to know” (vs. “nice to know”) 
questions. As the survey designer, you must 
understand why you are asking each question, 
how each question fits with the overall survey, 
and how the information will be used. Learning 
the concerns and perceptions of the respondents 
prior to survey design will provide significant 
value to the survey design and the ultimate 
outcomes of your evaluation. 

We highly recommend testing the survey 
items you have developed on a small sample. 

The clarity of the questions, the length of the 
instrument, and the nature of the questions 
(e.g., are they culturally sensitive or too 
personal?) can all affect whether the responses 
you get are both complete and accurate. 

Based on the test run, you can make necessary 
modifications before launching into full-scale 
data collection. 

While it is important that you start to develop 
your own survey instruments based on your 
evaluation needs, we have provided examples of 
survey instruments to assist you. Feel free to use 
these instruments in your own evaluations and to 
modify them to meet your specific evaluation 
needs. Note that we left blank lines and spaces 
where we thought you might want to insert your 
own items, statements, or questions. The first set 
of surveys is for evaluating specific activities or 
events that comprise your program. Information 
collected by these surveys can be used to modify 
and refine current activities and events and to 
design and develop new ones. The second set of 
surveys is for collecting information about 
program impact on participants. 

ERJC — 
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Sample Surveys for Evaluating 
Program Activities or Events 

Your program undoubtedly is comprised of a 
series of activities and events which may include 
field trips, workshops, seminars, observations, 
visiting speakers, etc. While all of these 
components can have an impact on participants 
in the aggregate, it is important to understand 
what participants think about individual 
activities and events while, or soon after, they 
engage in them. It is important to gather 
information about activities or events when they 
happen because it is difficult for participants to 
remember specific aspects of activities or events 
once too much time has passed, and the 
information you collect may be erroneous and 
unreliable. Evaluating individual activities and 
events not only provides you with information 
about their impact on participants, but also can 
serve as a formative tool for improvement along 
the way. In other words, it provides you with 
information on what worked, what didn't work, 
how the activity can be improved, etc. 

It is probably not necessary or even efficient 
to evaluate every activity/event. Instead, you 
should focus on the primary events/activities 
that comprise your program, on new events/ 
activities, on events/activities that you are 
considering altering or eliminating or on events/ 
activities for which you may be seeking funding. 
You will need to determine the components of 
your program that should be evaluated based on 
your program needs, resources, and long-term 
strategic planning. 

Evaluation of an activity or event entails 
collecting information across four main areas: 

• Instructional Delivery — including the 
presenter, the presentation, and 
participation; 

• Content — including focus, level, and 
relevancy; 

• Participant Benefit — including 
applications, insights, learning 
experiences, utility; and, 

• Overall Session — including 
organization, logistics, visual aids, 
expectations. 
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You can also use open-ended questions to elicit 
information about participants needs, including 
what questions they still have, what further 
information they would like, and suggestions for 
improving the activity or event. 

We have provided two surveys for use with 
your own programs: A Participant Satisfaction 
Survey which asks detailed questions about a 
particular activity or event, and a quick “3-2-1” 
Evaluation Form which is designed to gather 
quick responses regarding what participants 



learned from the activity or event and where 
they still need information and assistance. You 
may use these survey instruments as is or you 
may modify them to meet the needs of your 
own evaluation. Suggestions for the 
administration of these surveys follow. Also 
provided is an inventory of evaluation questions. 
Use this inventory in creating your own survey 
instruments, but also add to it and share it with 
others as you think of new survey questions. 



P Administration of Activity/Event Evaluation Questionnaires 


What: 


The following Participant Satisfaction Survey and “3-2-1” Evaluation 
Form are designed to gather information about specific activities or 
components of your program. The Participant Satisfaction Survey 
collects in-depth information about a particular event, including the 
quality of instruction, presentation, materials, logistics, etc. The “3-2- 1” 
Evaluation Form collects “quick response’ information about what 
participants learned and what they still want to know. 


Who: 


The surveys should be administered to ALL participants in a particular 
activity or event. ; Ck 4 . V'v 


When: 


If possible, the surveys should be completed by participants at the end of 
the event itself. 


Where: 


Try to get all participants to complete the surveys at the event before they 
leave. This will help to ensure a high response rate and will save vou the 
time and frustration of trying to track down participants later. 


HOW: 

f 


During the event or activity (usually at the start or at wrap-up), tell the 
participants that you will be asking them to complete an evaluation 
questionnaire about the event/activity before they leave for the day. Let 
them know that the information they provide will help you to determine 
the quality, relevance, and usefulness of the event/activity and that you 
will use the information to make adjustments to the event/ activity and 
to plan for other such activities. Stress that their input will help to 
improve the overall program and that it is very important that everyone 
respond to the survey. Indicate that no names will be used in any 
reporting, that you and your staff are the only ones to see the data, and 
that their confidentiality as respondents is ensured. Before distributing 
the survey, address any questions they may have. Have someone from 
the staff go around and collect the questionnaires to ensure that they 
beety completed and that no questionnaires are “floating about.” 
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£ Participant Satisfaction Survey 



ACTIVITY/EVENT 



DATE 



FIRST NAME 



LAST NAME 



Please take the time to complete this survey. It will only take a few minutes and will provide valuable 
feedback to us in planning future events and activities and in designing a program to meet your needs. 
Thank you for your cooperation! 



I. INSTRUCTIONAL DELIVERY 




Indicate which of the following apply to the presenter/instructor of this event: 



( check all that apply) 



A. The presenter was well-prepared 

B. The presenter was knowledgeable about the subject 

C. The presenter was a dynamic teacher 

D. The method of presentation held my attention 

E. The presentation was clear and easily understood 

F. The presenter answered all questions satisfactorily 

G. Audience participation was encouraged 

H. There was enough time for questions and discussion 



YES 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 



NO 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 



II. CONTENT 

Indicate which of the following apply to the content of this event: ( check all that apply) 



YES 

A. The level of content was appropriate O 

B. The information presented was useful for me O 

C. The presentation increased my knowledge of the topic □ 

D. The content was interesting to me O 

E. The content was relevant to my interests □ 



NO 

□ 

□ 

□ 

□ 

□ 



III. BENEFITS 

Indicate how this event/activity benefits you: ( check all that apply) 




0 



A. I learned new skills 

B. I can apply what I learned 

C. This activity/event was worth my time 

D. This activity/event accomplished what it said it would 
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YES 

□ 

□ 

□ 

□ 



NO 

□ 

□ 

□ 

□ 
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IV. OVERALL ACTIVITY/EVENT 

Indicate which of the following apply to this activity/event: ( check all that apply) 

YES NO 

A. This activity /event was well organized □ □ 

B. My expectations for this activity/ event were met □ □ 

C. I would recommend this activity/ event to others □ □ 

D. I would like to participate in other activities/events □ □ 

similar to this one in the future 



V. 0PEN-EN0E0 QUESTIONS 

A. How will you use the information and skills you learned today? 



B. What did you hope to learn from this activity/event, but did not? 



C. If you had one thing to say about this activity/ event, it would be: 



D. Please list other comments and suggestions for this activity/event: 
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% Inventory of “Participant Satisfaction” 
Survey Questions 



INSTRUCTIONAL DELIVERY OR METHODS QUESTIONS 

Knowledge 

1. The presenter demonstrated mastery of the topic 

2. The presenter was knowledgeable 

3. The presenter addressed the stated learning objective for the session 

4. The presenter was well-prepared 



Participation 

5. The presenter encouraged me to express my opinion or experiences 

6. I had the opportunity to participate in the discussion 

7. Audience participation was encouraged 

8. I had the opportunity to participate in discussion with my peers 

9. There was sufficient interaction among participants 




Presentation 

10. The presentation was clear 

1 1 . The presentation was to the point 

12. The method of presentation held my attention 

13. The main points of the presentation were clearly understood 



Relates to Audience 

14. The presenter was responsive to participants 

15. The presenter was able to relate to the ability level of the audience 

1 6. The presenter answered questions satisfactorily 

17. The presenter maintained the interest of the group 

18. The presenter broke down complex topics for easier explanation 



Speaking Skills 

19. The presenter was enthusiastic about the topic 

20. The presenter demonstrated good speaking skills 

21 . The presenter was a dynamic teacher 

22. The presenter initiated fruitful and relevant questions 

23. The presenters handouts, materials, etc. were valuable as learning aids 

24. Program handouts/materials enhanced the presentation 

25. The instructional materials used in this program were helpful 
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Organization 

26. The presentation was well organized 

27. The pace of the program was appropriate 

28. The learning environment stimulated idea exchange 

29. There was sufficient time to network with others 

30. The time provided was sufficient to accommodate the subject 

31. There was enough time allotted for questions and discussion 

CONTENT QUESTIONS 

Focus 

32. The content reflected the advertised information 

33. The content was consistent with its description 

34. The content was current 

35. The content had substance 

36. The program objectives were clear 

37. The program objectives were met 

Level 

38. The difficulty level was appropriate 

39. The content was appropriate 

40. The content was too basic 

41. The content was too advanced 

42. The material presented was of practical use 

Relevance 

43. The content was interesting to me 

44. The content was relevant to my work 

45. The topic was relevant to my interests 

BENEFIT QUESTIONS 

Application 

46. I can apply what I learned in my situation 

47. I learned new applications 

48. The information provided was useful to me 

49. I improved my ability to solve problems in my work 

50. This activity/event gave me skills directly applicable to my career 

Insights 

51.1 gained new insights relevant to me 

52. My understanding of key matters was enhanced 

53. My thinking about the topic is more focused 

49 
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Learning Experience 

54. I learned something new 

55- The presentation challenged my thinking 

56. My knowledge of the subject was enhanced 

57. I learned something useful from this presentation 

58. I learned a lot from this presentation 



Value 

59. The activity/event was worth my time 

60. My learning objectives were addressed 

61. The program accomplished what I thought it would 

62. I would recommend this to others 

63. This program met my expectations 

64. The choice of activity/event suited me 

65. This activity expanded my knowledge 

66. The activity was a good value for my money 



OVERALL SESSION QUESTIONS 




67. The activity/event was well organized 

68. The topics were relevant to my interests 

69. My objectives for this activity/event were met 

70. My expectations for this activity/event were met 

71. Overall, this activity/event was beneficial 

72. Overall, I would rate this activity/ event as: 



OPEN-ENDED QUESTIONS 




73. The best thing about this activity/event was: 

74. How did you learn about this activity/event? 

75- Why did you attend this activity/event? 

76. I need more information on: 

77. What part of the program did you find to be of particular benefit? 

78. What was the most useful information you learned from this program? 

79. What topics needed more time? Less time? 

80. What are your suggestions for future topics? 

81. What part of this program/activity/event was most useful to you? 

82. If someone were to ask you about this program/activity/event, you would say: 

83. This program/activity/event could be improved by: 

84. If you had one thing to say about this program/activity/ event, it would be: 

85- With whom will you share the information you collected today? Why? 

86. How will you use the information you collected today? 
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Evaluation 3-2-1 




things I learned from this activity: 



j|| things I am still confused about and would like to learn more about: 




thing I will do or try based on what I learned today: 



51 



Name: 



40 



Measured Steps 




Sample Participant Outcomes Surveys 

Ultimately, you want to know if your teacher 
recruitment program has an impact on its 
participants. The “outcome” you measure is 
determined by your evaluation purpose and its 
research questions. While the surveys on the 
previous pages help to collect information for 
addressing “process” research questions about 
specific program components (e.g., “Which 
activities were most/least effective?”), the next 
set of surveys will help you address “impact” 
research questions (e.g., “Are participants 
acquiring the skills and knowledge necessary to 
enter a college teacher education program?” 

“Did this program have an impact on 
participants’ post-high school plans and goals?”). 

In assessing the impact of your program on 
participants, it will be important to collect 
information that will allow you to examine 
change over time and to determine the influence 
of your program on participants as they move 



through school and pursue post-secondary 
plans, including college and teaching careers. 
Tracking students over several years is a difficult 
endeavor, often requiring extensive time, 
resources and expertise. A full-scale longitudinal 
study of participants through the program and 
beyond (which might include tracking students 
annually over several years) would probably 
require consultation with an external evaluator. 
There are, however, key data elements that 
should be collected from students on the survey 
form that can increase the chances of 
maintaining contact with them once they 
graduate or leave the program. These data 
include: full name, parent or guardian address, 
and social security number. 

Three sample surveys are provided in the next 
section to assistyou: ( 1 ) a Participant Tracking 
Survey, ( 2 ) a Non-Returning Participant Survey, 
and (3) an Alumni Survey. Suggestions for the 
administration of these surveys follow. 




Remember, surveys are 
only one way of 
collecting information 
to address your 
research questions and 
assess impact. Using 
other methods such as 
interviews and 
observations can help 
to strengthen and 
enhance the 
information collected 
through a survey. The 
more quality 
information you have, 
the stronger your case. 
Refer to the Resources 
section for books and 
references on a range 
of data collection 
methods. 



Administration of Participant Tracking Survey 



What: 



Who: 

When: 



Where: 

How: 



MC 



The Participant Tracking Survey consists of two forms — Part A to be administered when the 
participant enters the program and Part B to be administered when the participant completes the 
program. Coupled with other key information you should collect from participants (including GPA, 
school and/or program attendance, and standardized test scores), these surveys can be used to make 
assessments about the impact of the program on participants’ post-high school educational and career 
plans, attitudes about teaching, and preparation for teaching. Because this survey is designed to be 
used by a range of programs, you 11 notice that several of the questions have blanks for you to Fill in 
specific statements based on the unique components or activities of your program.: 

The surveys should be administered to ALL participants. ; £y:, . 4 . 

The surveys should be completed by participants when they enter the program and when they 
complete the program. (If a participant does not complete the program , he/she would also complete a 
“Non- Returning" Participant Survey— see next section.). If your program is already in operation, you 

may want to administer Part A to current participants as well as to new participants^^ 

* • :*•**•; • -V " v '■ 

All participants should complete the surveys at the program site. This will Help to ensure aLigh 
response rate and will save you the time and frustration of trying to track down participants. 

When participants enter the program (e.g., first day of the program), tell them that you will be asking 
them to complete a participant questionnaire now and when they have completed the program. Let 
them know that the information they provide will help you to determine the quality, relevance, and 
usefulness of the program and it will help, you to understand participants’ experiences inJtKe program. 
Tell them how you plan to use the information. Stress that their input will help to improve the overall 
program and that it is very important that everyone respond to the survey. Tell them that you will 
share with them results of the evaluation. Indicate that no names will be used in any reporting, that 
you and your staff are the only ones to see the data, and that the respondents’ confidentiality is 
ensured. Before distributing the survey, address any questions thev may have. You may want to walk 
through the survey with participants before they complete it to address any questions they may have 
about specific items. 

BEST COPY AVAILABLE 
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Participant Tracking Survey- Part A 



Participant Tracking Survey for:. 



(Write in the name of the program) 



Marking Instructions: <&> DO NOT SKIP any questions unless directed to do so. 

; <&> Pencil or pen can be used to complete this survey. 



Social Security Number: 



Last name 


First name 


Middle Initial 


Street 


City 


Zip Code 


Phone# (including area code) 






Please give the name of a parent or guardian who 


lives with you: 





Last name First name 



Background Information 



1. Are you: □ female □ male 

2. Which best describes your racial/ethnic 

group? 

□ American Indian □ Caucasian 

□ Asian/Pacific Islander □ Hispanic 

□ Black/African American □ Other 

3. What is your birth date? / / 

Month Day Year 

4. What is your current grade level? 

3. How many years have you participated 
in this program? 

□ This is my FIRST year in the program 
I have participated in this program for 
years 



Lifelong Learning Plans 



6. Do you expect to complete a high school 
education and graduate? 

□ Yes □ No 

7. What are your plans for the first year after 
graduating/leaving high school? ( check all 
that apply ) : 

□ I don t have any plans (skip to #9) 

A. □ Apprentice Training Program in {specify): 



B. □ Paid employment 

What occupation best describes the type of job 
you will seek your first year after leaving/ 
graduating from high school {specify): 
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c. □ 



College 

□ Community college 

□ Technical college 

□ Four-year college 



What area do you plan to study? 
□ I don’t know 



□ I plan to study: 
D. □ Military Service 



E. □ Other plans {specify):. 



8. How certain are you about your plans for the 
first year after high school? 

□ CERTAIN 

□ SOMEWHAT CERTAIN 

□ UNCERTAIN 




Indicate the people who have provided you 
with support and encouragement and have 
helped you with your plans for after high 
school ( check all that apply): 

□ Friends 

□ Parent(s)/guardian 

□ Brother(s)/sister(s) 

□ Teacher(s) 

□ My school counselor 

□ Staff from this program 

□ Another adult (coach, instructor, 
employer, relative) 



10. Do you plan eventually to become a teacher? 

□ Yes How certain are you about wanting to 
become a teacher? 

□ CERTAIN 

□ SOMEWHAT CERTAIN 

□ UNCERTAIN 

□ No How certain are you about NOT 
wanting to become a teacher? 

□ CERTAIN 

□ SOMEWHAT CERTAIN 

□ UNCERTAIN 

1 1 . Whether or not you want to become a 
teacher, what are some things that might 
keep you from becoming a teacher ( check all 
that apply): 

□ I don’t have the grades to get into college to 
become a teacher 

□ I don’t have the motivation or desire to teach 

□ I don’t have the confidence to teach 

□ I don’t have the support from family and 
friends to become a teacher 

□ I don’t have the support from my teachers 
and counselors to become a teacher 

□ Teacher’s salary 

□ Concerns about school discipline 

□ Concerns about school safety 

□ 

□ 

□ Other {specify): 



Attitudes About Teaching 



12. Indicate whether you agree or disagree with the following statements: 





agree 


disagree 


a. Teaching is a rewarding career 


□ 


□ 


b. Teachers get a lot of respect 


□ 


□ 


c. It is easy to teach 


□ 


□ 


d. I know the steps that need to be taken to become a teacher 


□ 


□ 


e. 


□ 


□ 


f. 


□ 


□ 


g- 


□ 


□ 


h. 


□ 


□ 



o 
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Teaching Skills 



13. The following is a list of critical skill and ability areas for teaching or any career. Check the box 
that best describes your current skill or ability level. 



Do you have this skill, knowledge or 
capability to try for your goal? 



Critical Skill and Ability Areas 


Little or not at all 


Some, not enough 


Solid base 


Brainstorming and problem solving 


□ 


□ 


□ 


Interacting and communicating effectively with others 


□ 


□ 


□ 


Having a solid knowledge of the core academic areas, including 
math, science, and English 


□ 


□ 


□ 


Being sensitive and relating to people of different backgrounds 


□ 


□ 


□ 


Thinking creatively and coming up with new ideas 


□ 


□ 


□ 


Contributing and working well in a team or group setting 


□ 


□ 


□ 


Managing money, time and other personal resources 


□ 


□ 


□ 


Leading and supervising others 


□ 


□ 


□ 


Using computers to complete tasks and projects 


□ 


□ 


□ 


Identifying goals (personal, career, educational) and steps to reach your goals 


□ 


□ 


□ 


Demonstrating self-confidence in yourself and your abilities 


□ 


□ 


□ 




□ 


□ 


□ 




□ 


□ 


□ 




□ 


□ 


□ 




□ 


□ 


□ 



14. What are the top four or five things that you hope to get out of participating in this 
program? 

l._ 

2 

3. 

4. 
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0 Participant Tracking Survey- Part B 

Participant Tracking Survey for: 

(Write in the name of the program) 

Marking Instructions: * DO NOT SKIP any questions unless directed to do so. 

Pencil or pen can be used to complete this survey. 



Social Security Number: 



Last name 


First name 


Middle Initial 


Street 


City 


Zip Code 


Phone# (including area code) 

Please give the name of a parent or guardian who lives with you: 





Last name First name 



Background Information 



1. Are you: □ female □ male 

2. Which best describes your racial/ethnic 

group? 

□ American Indian □ Caucasian 

□ Asian/Pacific Islander □ Hispanic 

□ Black/African American □ Other 

3. What is your birth date? / / 

Month Day Year 

4. What is your current grade level? 

5. How many years have you participated 
in this program? 

I have participated in this program for years 



Lifelong Learning Plans 



6. Do you expect to complete a high school 
education and graduate? 

□ Yes □ No 

7. What are your plans for the first year after 
graduating/leaving high school? (check all 
that apply): 

□ I don't have any plans ( skip to #9) 

A. □ Apprentice Training Program in {specify): 



B. □ Paid employment 

What occupation best describes the type of job 
you will seek your first year after leaving/ 
graduating from high school (specify): 
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□ Community college 

□ Technical college 

□ Four-year college 
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What area do you plan to study? 

□ I don’t know 

□ I plan to study: 

D. □ Military Service 

E. □ Other plans {specify): 

8. How certain are you about your plans for the 
first year after high school? 

□ CERTAIN 

□ SOMEWHAT CERTAIN 

□ UNCERTAIN 

9. Indicate the people who have provided you 
with support and encouragement and have 
helped you with your plans for after high 
school ( check all that apply): 

□ Friends 

□ Parent(s)/guardian 

□ Brother(s)/sister(s) 

□ Teacher(s) 

□ My school counselor 

□ Staff from this program 

□ Another adult (coach, instructor, 
employer, relative) 



10. Do you plan eventually to become a teacher? 

□ Yes "■* How certain are you about wanting to 
become a teacher? 

□ CERTAIN 

□ SOMEWHAT CERTAIN 

□ UNCERTAIN 

□ No How certain are you about NOT 
wanting to become a teacher? 

□ CERTAIN 

□ SOMEWHAT CERTAIN 

□ UNCERTAIN 

1 1. Whether or not you want to become a 
teacher, what are some things that might 
keep you from becoming a teacher {check all 
that apply): 

□ I don’t have the grades to get into college to 
become a teacher 

□ I don’t have the motivation or desire to teach 

□ I don’t have the confidence to teach 

□ I don’t have the support from family and 
friends to become a teacher 

□ I don’t have the support from my teachers 
and counselors to become a teacher 

□ Teacher’s salary 

□ Concerns about school discipline 

□ Concerns about school safety 

□ Other {specify): 



Attitudes About Teaching 



12. Indicate whether you agree or disagree with the following statements: 





agree 


disagree 


a. Teaching is a rewarding career 


□ 


□ 


b. Teachers get a lot of respect 


□ 


□ 


c. It is easy to teach 


□ 


□ 


d. I know the steps that need to be taken to become a teacher 


□ 


□ 


e. 


□ 


□ 


f. 


□ 


□ 


g- 


□ 


□ 


h. 


□ 


□ 
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Teaching Skills 



13 . The following is a list of critical skill and ability areas for teaching or any career. In Section I, 
check the box that best describes your skill or ability level. In Section II, check the box that best 
describes how this program helped to establish or improve each skill. 



Section I 

Do you have 
this skill, 
knowledge or 
capability to 
try for your 
goal? 



Section II 

Extent that this 
program helped 
to establish or 
improve your 
knowledge and 
skills: 



Critical Skill and Ability Areas 


Little or not at all 


Some, not enough 


Solid base 


Not at all 


Somewhat 


Significantly 


Brainstorming and problem solving 


□ 


□ 


o 


□ 


□ 


□ 


Interacting and communicating effectively with others 


□ 


□ 


□ 


□ 


□ 


o 


Having a solid knowledge of the core academic areas, including 
math, science, and English 


□ 


□ 


□ 


□ 


□ 


o 


Being sensitive and relating to people of different backgrounds 


□ 


□ 


□ 


□ 


□ 


□ 


Thinking creatively and coming up with new ideas 


□ 


□ 


□ 


□ 


□ 


□ 


Contributing and working well in a team or group setting 


□ 


□ 


□ 


□ 


□ 


□ 


Managing money, time and other personal resources 


□ 


□ 


□ 


□ 


□ 


□ 


Leading and supervising others 


□ 


□ 


□ 


□ 


□ 


□ 


Using computers to complete tasks and projects 


□ 


□ 


□ 


□ 


□ 


□ 


Identifying goals (personal, career, educational) and steps to reach 
your goals 


□ 


□ 


□ 


□ 


□ 


□ 


Demonstrating self-confidence in yourself and your abilities 


□ 


□ 


□ 


□ 


□ 


□ 




□ 


□ 


□ 


□ 


□ 


□ 




□ 


□ 


□ 


□ 


□ 


□ 




□ 


□ 


□ 


□ 


□ 


□ 




□ 


□ 


□ 


□ 


□ 


□ 
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14. For each program component, activity, or attribute listed below, circle the grade you would give, 
from A to F, based on your overall experience in this program. (A= Highest rating; F=Lowest rating) 



Program Components, Activities, and Attributes 


What grade would you give this component of 




the program? (Circle only 


one 


grade per statement) 


1 . 


A 


B 


c 


D 


F 


2. 


A 


B 


c 


D 


F 


3. 


A 


B 


c 


D 


F 


4. 


A 


B 


c 


D 


F 


3. 


A 


B 


c 


D 


F 


6. 


A 


B 


c 


D 


F 


7. 


A 


B 


c 


D 


F 


8. 


A 


B 


c 


D 


F 


9. 


A 


B 


c 


D 


F 


10. 


A 


B 


c 


D 


F 


11. 


A 


B 


c 


D 


F 


12. 


A 


B 


c 


D 


F 


13. 


A 


B 


c 


D 


F 



15. What are the top four or five things (if any) that you got out of participating in this program 
and that you will use in pursuing your educational and professional plans and goals? 



1 , 

2 , 

3, 

4, 
3, 



16. What had you hoped to get out of this program (e.g., skills, knowledge, tools, support), but did 
not (if any)? 




1 , 

2 . 
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17. What are the top three things you would change about this program (if any)? 



1. _ 

2 , 

3, 



18. Have you recommended this program to other students? 



□ Yes 



□ I have not, but I would if the opportunity arose □ I have not, and I would not 



Other Comments: 





O 
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Administration of Non-Returning Participant Survey 




What: 

Who: 

When: 

Where: 

How: 




The Non-Returning Participant Survey is used to understand why 
participants leave the program (before completing the program) and 
provides critical information for issues of recruitment, retention and 
continued program development: Because this survey is designed to be 
used by a range of programs, you’ll notice that several of the questions 
have blanks for you to fill in specific statements based on the unique 
components or activities of your program. 

This survey should be administered to ALL participants who leave/don’t 
return to the program. 

The surveys should be completed by participants shortly after they leave 
the program and/or at the point you know they will not or did not 
return to the program. 

Ideally, non-returning participants would complete the survey at the 
program site (during an exit interview, for example). If that is not 
possible, you will need to determine the best way to reach the student. 
For example, if the student is still at the school, you could work with 
his/her homeroom teacher to get the survey to the student, or you could 
mail the survey to the students home with a cover letter describing the 
survey and how the information will be used. 

[See “Where” above] Whatever way you reach non-returning 
participants, let them know that the information they provide will help 
you to understand participants’ experiences in the program and that this 
is their opportunity to voice their concerns/issues with the program. 

Tell them how you plan to use the information (e.g., you will use the 
information to understand how well the program is working as well as to 
understand what parts of the program need to be improved). Stress that 
their input will help to improve the overall program and that it is very 
important that they respond to the survey. Indicate that no names will 
be used in any reporting, that you and your staff are the only ones to see 
the data, and that the respondents’ confidentiality is ensured. If it is a 
mail survey, make sure to provide this information in a cover letter and 
provide a telephone number where they can reach you with any 
questions they may have. 
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0 Non-Returning Participant Survey 



Non-Returning Participant Survey for: 

(Write in the name of the program) 





Social Security Number: 



Last name 


First name 


Middle Initial 


Street 


City 


Zip Code 



Phone # (including area code) 



Background Information 



1. Are you: □ female O male 

2. Which best describes your racial/ethnic 
group? 

□ American Indian □ Caucasian 

□ Asian/Pacific Islander □ Hispanic 

□ Black/African American □ Other 




7. Do you feel you are back on course and have 
either resolved or adapted to these issues? 

□ No □ Yes 

8. Did you seek assistance from any program 
staff in working to resolve these issues? 

□ No □ Yes 



3. What is your birth date? / / 

Month Day Year 

4. What is your current grade level? 

5. How many years have you participated 
in this program? 

I have participated in this program for years 



6. While in this program, did any of the 
following interfere with your progress or 
experiences, (e.g., missed activities; could not 
complete assignments, etc.? ( check all 
that apply): 

□ No —skip to # 9 

□ Personal motivation/self-discipline 

□ Academically under-prepared 

□ Physical disability/ health problems 

□ Depression/psychological problems 

□ Family- related problems 

□ Other {specify): 



9. Why did you decide not to return to this 

program? {check all that apply) 

□ I decided I didn’t want to become a teacher 

□ The time and/or location of the program 
activities were not convenient for me 

□ I had too many other obligations (e.g., my 
studies, a job, extracurricular activities) 

□ I had a personal/family- related problem to 
resolve 

□ I could not resolve a personal conflict with 
program staff 

□ I could not resolve a personal conflict with 
other students in the program 

□ The program activities were too basic 
for me 

□ The program activities were too advanced 
for me 

□ Other {specify): 
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: 10. For each program component, activity, or attribute listed below, circle the grade you would give, 
from A to F, based on your overall experience in this program. (A= Highest rating; F= Lowest rating) 



Program Components, Activities, and Attributes 


What grade would you give this component of 
the program? (Circle only one grade per statement) 


1 . 


A 


B 


C 


D 


F 


2. 


A 


B 


C 


D 


F 


3. 


A 


B 


C 


D 


F 


4. 


A 


B 


C 


D 


F 


5. 


A 


B 


C 


D 


F 


6. 


A 


B 


C 


D 


F 


7. 


A 


B 


C 


D 


F 


8. 


A 


B 


C 


D 


F 


9. 


A 


B 


C 


D 


F 


10. 


A 


B 


C 


D 


F 


11. 


A 


B 


C 


D 


F 


12. 


A 


B 


C 


D 


F 


13. 


A 


B 


C 


D 


F 



11. What had you hoped to get out of this program (e.g., skills, knowledge, tools, support), but did 
not (if any)? 



1 , 

2 .. 

3, 



12. What are the top three things you would change about this program (if any)? 

1 . 

2 

3. 



13. Have you recommended this program to other students? 

□ Yes □ I have not, but I would if the opportunity arose □ I have not, and I would not 



O 
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Administration of Alumni Survey 



Wliat: Alumni Survey: While it is important to examine the change in students while they 

are actively involved in the program, it is also important to examine what happens to 
students once they graduate/complete the program. Alumni surveys are used to 
' accomplish just that. While it is sometimes difficult to track down students once they 
'' rm ^CSSe the program, by collecting information up-front, like their full name, phone 

number, and parent/guardian address, you will be more successful in program follow- 
up. The alumni survey provided here focuses on a high school pre-collegiate teacher 

recruitment program and assumes the recipients of these surveys would be high school 
■graduates. Because this survey is designed to be used by a range of programs, you’ll 
notice that several of the questions have blanks for you to fill in specific statements 
based on the unique components or activities of your program. 



The surveys should be administered to participants who have graduated from the 
program. Because it is unlikely that you have a very large number of participants 
from each “graduating class,” you will want to survey all participants from a 
graduating class and make sure that you receive the survey back from as many 
participants as possible and that the respondents are representative of the group as a 
whole. For example, if you have a graduating class of 20 participants and 10 are male 
and 10 are female, then you will want to make sure to have a similar ratio of 
respondents. If you have 10 respondents and they are all female, you may make some 

assumptions about the program based solely on the female perspective. It is 

important, therefore, that you work to ensure that various participant types (e.g., sex, 
race) are represented adequately in the surveys returned. Of course, the higher the 
number of respondents to return the survey, the greater the likelihood that all 
participants are represented. 

When: Based on your evaluation purpose and research questions, you will need to determine 

which graduating classes you want to survey. You may survey recent alumni (e.g., 
those who recently completed the program and graduated from high school) or you 
may be interested in what alumni of the program are doing five years after graduating. 

Where: 'TYou will need to determine the best way to reach program alumni based on your time . 

?and resources “ Since you are dealing with alumni, you will most likely have to maiL.^ 

\ • ■ ; lithe surveys to past participants or have them complete the survey over the phone. gg-.. 

This is where the tracking information you collect (name, parent address, phone 
number) comes into use. For example, you can mail the survey to the participant^ .y 

parent address and ask the parents to forward the survey to the participant. On the 

survey itself, you ask for the participants current address and can then. use | ,that,rc> . 
update your database for future mailings. 



WhO: 

0 



HOW: 




;:[See “Where ’ above] Whatever way you reach non-returning participants, let.them^/ 
know that the information they provide will-help you to determine and to understand 
participants’ experiences in the program and how it relates to their post-program , ^ 
plans and goals. Tell them how you plan to use the information (e.g., to understand 
how well the program is working; to assess what parts of the program need to be 
improved, etc.). Stress that their input will help to improve the overall program and 
that it is very important that they respond to the survey. Indicate that no names wil 
be used in any reporting, that you and your staff are the only ones to see the data, and 
that the respondents' confidentiality is ensured. If it is a mail survey, make sure to 
provide this information in a cover letter and provide a telephone number where they 
can reach vou with any questions they may have. 
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Alumni Survey 



Alumni Survey for 



(Write in the name of the program) 



Marking Instructions: v |^|D0 NOT SKIP anpuestions unless directed^) do so. 

1 ; \ changes..you make: = ^; ■■■ : 

to complete this survey.'-= : " 



Social Security Number: . 



Last name 




First name Middle Initial 


Street 




City Zip Code 


Phone # (including area code) 






Background Information 




|| Current Status 


1 . Are you: □ female 


□ male 


7. Are you currently pursuing a post-secondary 






degree? 


2. Which best describes your 


racial/ethnic 


□ No 


group? 




□ Yes ,m + indicate the degree you are currently 


□ American Indian 


□ Caucasian 


pursuing: 


□ Asian/Pacific Islander 


□ Hispanic 


□ Certificate in: 


□ Black/African American 


□ Other 








□ Associates degree in: 


3.What is your birth date? _ 


/ / 


□ Bachelors degree in: 


Month Day Year 


□ Masters degree in: 






□ Doctorate (Ph.D. /Ed.D) in: 


4. In what year did you start 


this program? 


□ Professional degree (Law, Medical) 


199 




in: 



5. In what year did you complete this program? 

199 

6. Which best describes your highest 
educational attainment to date ? ( check the 
highest degree that you have completed ) 

□ High school diploma or less 

□ Some college courses but don’t have a 
degree 

□ Associate Degree 

□ Bachelors Degree 

□ Masters Degree 

□ Professional degree (MD, JD) 

□ Ph.D., Ed.D 

65 



8. What is your CURRENT employment 
situation? ( check only one) 

□ Employed with one job 

□ Employed with two or more jobs at same 
time 

□ Apprentice Training Program / Internship 

□ In military (go to #10) 

□ Homemaker (go to #10) 

□ Unemployed (looking for work) 

(go to #10) 

□ Not employed, other (go to #10) 

BEST- COPY AVAILABLE 
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9. Is your primary occupation as a teacher? 

□ No what is your occupation title? (specify): 



Attitudes About the Program 



D Yes what grade level(s) do you teach? 

□ pre-K 

D elementary (K-5) 

D middle/junior high (6-8) 

□ high school (9-12) 

D post-secondary (college) 

10. In what field/ occupation do you ultimately 
want to work? 

D None, do not plan to work 

D Same occupation indicated in Question 9 

□ Other occupation 

List the field/occupation in which you 
ultimately would like to work: 



1 L Did the pre-collegiate teacher recruitment 
program listed in the title of this survey 
influence your post-high school plans? 

□ No 

□ Yes in what ways? ( check all that apply): 

D Encouraged me to attend college 

D Helped me understand the steps I needed 
to take to get into college 

□ Prepared me with the academic skills to get 
into and succeed in college 

□ Provided me with practical experiences 
that helped me understand career options 

D Helped me obtain information about 
financial aid needed to attend college 

n 

D Other (specify): 



12. Overall, rate this program in terms of its 
influence in helping you with post-high 
school plans: 

□ It was a primary influence 

□ It was an influence, but not a primary one 
D It was not an influence in helping me 

with post-high school plans 
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13. In column A of the chart below, indicate the level of importance of the following skills in your 
everyday life, including your work and educational activities. In column B, indicate the extent 
to which this program helped to establish or improve this skill. Answer Column B even if the 
skill is not important to your current life activities. 



Critical Skills 


A. How important is 
this skill in your current 
life activities? 


B. Extent that this 
program helped to 
establish or improve 
this skill: 




Not at all important 

Somewhat important 
Very important 


Not at all 

Somewhat 

Significantly 


Brainstorming and problem solving 


□ □ □ 


□ □ □ 


Interacting and communicating effectively 
with others 


□ □ □ 


□ □ □ 


Having a solid knowledge in the core academic 
areas, including mathematics, science, 
and English 


□ □ □ 


□ □ □ 


Being sensitive and relating to people 
of difference backgrounds 


□ □ □ 


□ □ □ 


Thinking creatively and coming up 
with new ideas 


□ □ □ 


□ □ □ 


Contributing and working well in a team 
or group setting 


□ □ □ 


□ □ □ 


Managing money, time, and other 
personal resources 


□ □ □ 


□ □ □ 


Using computers to complete tasks and projects 


□ □ □ 


□ □ □ 


Leading and supervising others 


□ □ □ 


□ □ □ 


Identifying goals (e.g., personal, educational, 
career) and the steps to reach your goals 


□ □ □ 


□ □ □ 


Demonstrating self-confidence in yourself 
and your abilities 


□ □ □ 


□ □ □ 




□ □ □ 


□ □ □ 




□ □ □ 


□ □ □ 




□ □ □ 


□ □ □ 




□ □ □ 


□ □ □ 




□ □ □ 


□ □ □ 
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14. Aside from the skills/abilities listed above, in what other ways has this program impacted you 
and your post-high school plans and goals? 



15. Knowing what you now know, and looking back on your experience in this program, in what 
ways could we improve the activities, services, and support provided through this program? 




16. Overall, if you had one thing to say about this program, it would be: 
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Technology TIP: While 
you can do some analysis 
of your data without the 
use of computer 
software, as you collect 
more data over time, it 
will become increasingly , 
difficult to analyze the 
information by hand. It is 
therefore recommended 
that you store the data in 
an information database 
that can also be used for 
data analysis. If you are 
not familiar with data 
entry, coding, and 
analysis using a 
computer, you should 
seek out the assistance 
of someone who has 
experience in this area. 
For example, within a 
school district you may 
want to contact a 
technology teacher or a 
programmer at the 
district office. Graduate 
students from local 
colleges or universities 
are often available for 
assisting in data analysis. 
O 




0. K., What Does All of This Mean? 

At this point, you may have two types of data 
collected through various collection procedures: 
qualitative and quantitative. As pointed out 
before, qualitative data are usually collected 
through a narrative description and often provide 
in-depth information about the topic of study; 
quantitative data are data represented numerically. 
Regardless of the type of data you have 
collected, one thing is clear — you now have to 
make sense of it all. 

The basics for data analysis and 
interpretation involve at least three key steps: 

• Checking the raw data and preparing 
them for analysis; 

• Conducting an analysis of the data 
based on your research questions; 

• Drawing interpretations of what the 
data mean based on your analysis. 

A basic overview of these steps is presented 
below and is based on the analysis of data collected 
through a participant survey that includes both 
closed-ended and open-ended questions. 

1 . Checking the raw data and preparing 
them for analysis 

The first step in data analysis is to “eyeball” your 
data through data checking for responses that 



Original Item 



11. Whether or not you want to become a 

teacher, what are some things that might 

keep you from becoming a teacher? 

( check all that appty)\ 

□ I don t have the grades to get into college 
to become a teacher 

□ I don t have the motivation or desire 
to teach 

□ I don t have the confidence to teach 

□ I don’t have the support from family 
and friends to become a teacher 

□ I don’t have the support from my teachers 
and counselors to become a teacher 

□ (etc.) 



may be out of line, inconsistent, or unrealistic. 
Examples might include checking off more than 
one response when only one can be selected, 
skipping questions that should have been 
answered, giving responses (to open-ended 
questions) that are not related to the question, and 
giving inconsistent responses. Depending on how 
out-of-line or unlikely these responses are, it may 
be necessary to eliminate these items from the data 
to be analyzed or to eliminate a respondent from 
the pool of respondents. For example, if responses 
to an item seem to be out-of-line across all 
respondents, then it may just be a “bad” item (e.g., 
respondents did not understand the item, the 
phrasing of the item was off, etc.) and you should 
consider dropping that item from the analysis. If 
you find that the responses from an individual 
respondent are out-of-line across the majority of 
the items, you may decide to drop that individual 
from the analysis. In both cases, keeping unreliable 
information in the analysis would lead to 
misinterpretations of the results. 

Once you have checked your data, you will 
want to decide on a format for analyzing the 
data. This will depend on the computer software 
you will use to enter and store the data. You will 
also need to code your data in numerical form 
for quantitative analysis. If, for example, the 
survey you used is “pre-coded” — respondents are 
asked to select an item from a list — then coding 
is relatively easy. You would simply need to 
replace the check marks with a number. Pre- 
coded item #11 from the sample Participant 
Survey provided in this handbook would be 
coded as follows: 



Coded Item 



11. Whether or not you want to become a 
teacher, what are some things that might 
keep you from becoming a teacher? 

( check all that apply ) : • \ ; TV 

1. I don’t have the grades to get into college 

to become a teacher r . ■= ^ : 

2. I don’t have the motivation or desire 
to teach 

3. I don’t have the confidence to teach 

4. I don’t have the support from family 
and friends to become a teacher 

5. I don’t have the support from my teachers 

and counselors to become a teacher 

6. (etc.) 
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Once you have coded all of your data, you 
would then key the data into a computer 
software program that could store and analyze 
it. 

Data cleaning includes a final check of 
the data file for accuracy in the coding of the 
data and in entering it into a computer 
program. 

2. Conducting an analysis of the data based 
on your research questions 

Once you are comfortable with the data 
checking and preparation, the next step is to 
conduct an analysis of the data based on your 
evaluation plan and your research questions. 

In conducting this analysis, you will 
undoubtedly need to use some statistical 
concepts and measurements. While some 
forms of statistical analysis can be complex 
and, at times, intimidating, there are some 
fairly simple techniques that can provide 



Research Questions 



|| l| | !!|| ill! II | llllliHl 1 Hi llllllllllll |i 

1. Does the program serve types of students 
most underrepresented in the field of 
teaching? 



2. Does the program attract students who are 
on a Teaching Track? 



valuable information. These techniques will 
be described here, while the Resources section 
of this handbook provides references to 
textbooks that outline other types of 
statistical methods. 

Descriptive Statistics 

Statistical procedures used in describing the 
properties of your sample are often referred 
to as descriptive statistics . Many evaluation 
questions can be answered through the use 
of descriptive statistical measures. Common 
descriptive statistical measures include 
frequency distributions (e.g., how many cases 
fall into a given category); means (average) 
and medians of responses; and range of 
responses. Some examples of these measures, 
based on the Participant Survey , are 
presented below. 

Let’s assume that your evaluation 
included the following research questions: 



A. What is the racial composition of 
participants? 

B. What is the composition of participants 
' by sex? 

A. What percent of participants expect 
to graduate from high school? 

B. What percent of participants expect 
to go to college? 

C. What percent of participants expect 
to become teachers? 



Sub-Questions 



3. What are the barriers that may keep 
students from entering the teaching field? 



A. What is the average number of barriers 
indicated by students? 

B. What is the most commonly checked 
barrier? 
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While the focus here is on 
providing you with some 
basic statistical methods 
for describing your data, 
there exists another form 
of statistics called 
inferential statistics. In 
simple terms, inferential 
statistics are used to 
examine response 
differences between two 
or more groups {e.g., 
males and females) and 
determining if the 
differences are real or 
“significant." Inferential 
statistics include things 
like t-tests, analysis of 
variance, and regression 
and while most statistical 
software covers these 
methods of analysis, 
understanding how to use 
them and what their 
information represents 
requires some level of 
statistical knowledge. 
While we do not cover 
inferential statistics here, 
we have suggested some 
textbooks in the 
Resources section for 
your review. 
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TECHNOLOGY TIP: The 
quantitative data you 
| have collected, once 
coded and entered into a 
' computer, can be 
analyzed by the 
computer using various 
programs. Today, 
statistical software (e.g., 
SPSS or SAS) is 
| sophisticated enoughtqfi| 
read data from multiple 
formats, including Text 
Database, and 
Spreadsheet. Again, we 
suggest that if you are 
not familiar with using a 
i computer for data entry 
: or analysis, you should 
seek out the assistance j 
\ of someone who has 
experience in this area, 
such as a technology 
teacher, a programmer at 
the district office; or SirS 
graduate students from 
local colleges or 

... f.tj 

rersities. 
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In addressing these questions, some very 
basic descriptive statistics can be employed: 
Frequency Distributions Lets assume 
that all 50 of your program participants 
completed the Participant Survey — Part A. 

To answer research question #1 from your 
computer using statistical or spreadsheet 
software, you would run a frequency 
distribution on questions #1 and #2 of the 
survey (racial and gender groups). The 
computer program would group your data. 
The output would look similar to the 
following graph ( illustration only): 



Racial Group 


Number of 
Participants 


Percent of 
Participants 


American Indian 


i 


2% 


Asian/Pacific Islander 


2 


4% 


African American 


20 


40% 


Caucasian 


12 


24% 


Hispanic 


15 


30% 


Other 


0 


0% 


TOTAL 


50 


100% 



Sex 


Number of 
Participants 


Percent of 
Participants 


Male 


10 


20% 


Female 


40 


80% 


TOTAL 


50 


100% 



A bar graph is another way to display these 
data: 



Distribution of Participants by Sex 




■ Male 

■ Female 



In addressing research question #2 ( Does 
the program attract students who are on a 
Teaching Track?) > you could run frequencies 
on the three sub-questions: percent that plan 
to graduate from high school, percent that plan 
to go to college, and percent that plan to teach . 
A frequency table would look something like 
this (illustration only): 





No. of participants 
who said “yes" 


• Do you plan to graduate 
from high school? 


48 


48/50 (96%) 


• Do you plan 
to attend college? 


40 


40/50 (80%) 


• Do you plan to 
become a teacher? 


30 


30/50 (60%) 



A bar graph is another way to display these 
data: 



Percent of Participants Responding “Yes” 




■ Will graduate 
M Will attend college 
□ Will teach 



Calculating Means In answering research 
question #3 ( What are the barriers that may 
keep students from entering the teaching 
field r), you could use statistical software to 
calculate means on the survey question that 
asks about barriers to becoming a teacher 
(Item 11 — there are nine barriers for 
respondents to check). Let's say that you 
wanted to know the mean (average) number 
of barriers that participants checked off and 
the raw data looked as follows ( illustration 
only): 



# of barriers checked: 


1 


2 


3 


4 


5 


6 


7 


8 


9 


# of respondents: 


15 


15 


5 


5 


5 


0 


5 


0 


0 
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Statistical software enables you to run 
means and frequencies relatively easily, but if 
you do not have access to such tools, you can 
easily use pencil and paper to calculate the mean 
number of variables checked off by respondents 
directly from your raw data. In this case, the 
mean (average) is 2.8. [because 140 barriers + 
50 respondents = 2.8]. 

Calculating the Median Using the raw data 
displayed on the previous page, you can 
calculate the median number of barriers checked 
(i.e., the value such that half the observations 
fall above it and half below it) as follows: 



Number of 
Barriers Checked 


Number of Respondents 
(* = 1 Respondent) 


1 


*************** 


2 


********** | ***** 


3 


***** 


4 


***** 


5 


***** 


6 




7 


***** 



In this case, with 50 observations, the 
median number of barriers would be between 
the 25 th and 26 th observations, which is 2.0. 

Looking at Change Over TimeThe primary 
reason for surveying participants when they 
enter the program and again when they 
complete the program is to examine how their 
attitudes, knowledge, behaviors, etc. have 
changed over time and to assess the degree to 
which the program contributed to those 
changes. Remember that you can never make a 
direct causal linkage between the program and 
changes in participants. In other words, you can 
never say with 100% certainty that it was the 
program that caused the change. You can, 
however, infer that the program contributed to 
changes in participants and assess the strength of 
that contribution relative to other factors that 
may have contributed to the changes as well. 

This form of analysis is referred to as inferential 
statistics and includes things like t-tests, analysis 
of variance, and regression (see previous Tip Box). 



For example, lets say you were interested in 
looking at the difference between participants 
when they entered the program and when they 
completed the program on the percent that 
planned to go on to college. Your data showed 
that at the start of the program, 20 percent of 
the participants planned to go to college and at 
the end, 80 percent planned to go to college. Is 
this a significant difference? Eyeballing the data, 
we would say it is. But this doesn’t tell you if it 
is a statistically significant difference. To know 
that, you would have to use inferential statistics, 
specifically a t-test. While we do not cover the 
details of inferential statistics here, you can refer 
to the Resources section for a bibliography of text 
books that explain manipulating statistics for 
examining such questions. Although you 
couldn’t say with 100% certainty that it was the 
program that caused this change, using 
inferential statistics you could say that after 
attending the program there was a statistically 
significant increase in the number of participants 
who planned to go to college and that the 
program most likely contributed to this change. 

If you choose not to use inferential 
statistics, however, you can still report on and 
describe a change in program participants over 
time. What you couldn’t do is make statements 
about statistical significance. What you could 
do, however, is support this interpretation by 
including other data collected, such as quotes 
from interviews with participants or participant 
testimonies about the program and its impact 
on their plans. 

Qualitative Data Analysis 

As mentioned before, you can have two types of 
data, qualitative and quantitative. Qualitative 
data are most likely collected through 
interviews, focus groups, observations, and 
open-ended survey questions. The analysis of 
qualitative data depends on the nature of the 
data, your research questions, and the purpose 
of your evaluation. 

Methods for analyzing qualitative data 
range from preparing narrative descriptions to 
performing quantitative analyses of narrative 
concepts. For example, you can actually code 
the number of time respondents mentioned 




Once again, for a more 
detailed discussion of 



methods, we refer you to 
the Resources section. 
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certain words, phrases, or concepts and then 
turn that into a descriptive analysis, looking at 
frequencies or means (averages). 

Searching for patterns and categories is 
another part of qualitative data analysis and 
once you have selected specific categories, 
patterns, or themes to support your research 
questions, you can use illustrative quotes from 
your data to support your analysis. Say you were 
interested in understanding how the program 
could be improved. You could, for example, 
analyze item 17 from the Participant Survey 
(Part B) which asks, “What are the top three 
things you would change about this program (if 
any)?” If you found specific comments repeated by 
many respondents, such as “more opportunities to 
mentor students,” that would constitute a pattern. 

3. Methods for Interpreting Evaluation Information 

The analysis of your data depends heavily on 
organizing and reducing information and then 
describing through quantitative and/or 
qualitative methods (as discussed above). 
Interpretation involves attaching meaning to this 
information and drawing conclusions. You can 
now integrate your tables, graphs and figures 
based on the analysis of the information, which 
will provide the basis for your final report. 

Quality interpretation involves careful 
methods of inquiry. For example, a responsible 
evaluator would never simply state, “The 
numbers speak for themselves.” You should be 
suspicious of anyone who boils interpretation 
down to such superficial statements! 

Interpretation, of course, is influenced by 
the evaluators own bias, perspectives, and 
experiences. The bottom line, however, is that 
results must be interpreted so that you and 
others will know how best to understand and to 
use the information. 

The field of evaluation is just beginning to 
develop systematic methods of interpretation, 
and new methods continue to be generated. 
Below is a list of some methods for interpreting 
evaluation information (Worthen & Sanders, 
1987): 

1. Determining whether objectives of the 
program have been achieved; 

- 73 — 



2. Determining whether assessed needs 
have been reduced; 

3. Determining the value of 
accomplishments; 

4. Asking critical reference groups to 
review the data and to provide their 
judgments of successes and failures, 
strengths and weaknesses; 

5. Comparing results with those reported 
by similar organizations or programs. 

While the evaluator is the primary 
interpreter of the evaluation information, he or 
she is not the only one. The evaluator/ 
evaluation team brings only one of many 
pertinent perspectives to bear. In fact, at times 
the evaluator may be too close to the evaluation 
and its data and needs to step back and receive 
input from others in order to see it more clearly. 
It is important therefore to interpret data 
through a “fresh set of eyes” and through 
multiple perspectives. 

One method of bringing multiple 
perspectives to the interpretation task is to use 
your stakeholders. Convening small groups of 
people with a direct interest in your evaluation 
(See Step 2: Identifying Audiences for Your 
Evaluation) to discuss their thoughts and 
interpretations of the evaluation information 
collected and analyzed is an effective way of 
understanding your data. It also gives 
stakeholders a “voice” and provides a forum 
through which stakeholders can, in essence, 
“buy-in” to the evaluation and its results. It is 
not necessary, however, to bring each and every 
stakeholder into the interpretation process. In 
selecting stakeholders to help with the 
interpretation of results, you should focus on 
people most directly linked to the program who 
provide useful insights and assistance, such as 
other staff and friends of the program. Before 
conducting your stakeholder meeting, you 
should supply stakeholders with the evaluation 
results, along with other important information 
such as the evaluation design and purpose and 
research questions. The meeting time can then 
be spent reviewing and interpreting evaluation 
results. Guiding questions to use in facilitating 
the interpretation process include: 
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What is the most salient information to 
present? 

What does this information mean? 

What is the practical significance of this 
information? 

Is it good, bad or neutral? 

What are the consequences and 
implications? 

What, if anything, should be done? 

How much data should be presented to 
support a conclusion? 

What should be done with inconsistent 
or contradictory information? 

These stakeholder interpretation meetings 
capture the range of perspectives and points-of- 
view, and also contribute to the usefulness of the 
evaluation while ensuring that those who should 
be involved are. 



Worksheets #6A and #6B 



Determining Your Analysis Plan 

and Interpreting Your Results 

Make a copy of these worksheets for each member of 
the evaluation design team. As a group and based on i 
information from Worksheets #4 & #5, identify your 
analysis plan as well as ways to interpret the 

information you have analyzed. 
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Worksheet #6A: Determining Your Analysis Plan 
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Worksheet #6B: Interpreting Your Results 




Drawing Conclusions 
and Making 
Recommendations 



Whew! You’ve come a long way in evaluating the 
program! Now that you’ve collected, analyzed, 
and interpreted your data based on your 
research questions, it’s time to draw some 
conclusions and make recommendations for 
moving forward. 

Drawing Conclusions 

Going back to the beginning of this handbook, 
you’ll recall that evaluation is about making 
judgments concerning the quality, effectiveness, or 
value of something — in this case, your teacher 
recruitment program. To make a judgment about 
the effectiveness, usefulness, or “goodness” of 
something, you must first apply predetermined 
criteria or standards against the data. Criteria or 
standards are what you and your colleagues believe 
to be appropriate levels of progress toward meeting 
previously identified program goals and/or 
participants’ needs. For example, let’s say that one 
of your research questions is, “How well does this 
program recruit minority students?” To answer 
this question and make conclusions about your 
program based on the evaluation data collected, 
you would need to have a standard against which 
to judge the data. If, for example, you used a 50 
percent increase in the number of minority 
students entering the program from Year 1 to Year 
2 as the standard, and your data showed that you 
got a 100 percent increase in the number of 
minority students enrolling in the program, then 
you could draw specific conclusions about the 
effectiveness of this program in recruiting minority 
students based on this standard. And one 
conclusion would be that a strength of the 
program is recruiting minority students. 

Once you have identified the standards or 
criteria you will use to make judgments about 
the quality of what has been evaluated, the next 
step is to make conclusions about your program 
based on the data you’ve collected and analyzed. 
One common way of organizing your 



conclusions or judgments is under the headings 
of program strengths and program limitations (or 
weaknesses ). Organizing your conclusions this 
way is straightforward, easily identifiable to a 
range of audiences, and leads easily to your 
recommendations. 

Making Recommendations 

Recommendations for program improvement 
should flow from your conclusions and be based 
on the purpose of your evaluation and its 
research questions. In thinking about 
recommendations for your program, you can 
use the organizing headings (e.g., strengths and 
limitations) as a guide. Another effective way of 
organizing recommendations is by “short” term 
(e.g., over the next few months) and “long” term 
(e.g., within the year) goals for program 
improvement. 

Often, external evaluators will include 
recommendations for the client to consider in 
making improvements to the program and these 
recommendations are or are not included in the 
final report, depending on their nature (e.g., 
internal recommendations about organizational 
structure may not need to be shared with 
external audiences). Because you are evaluating 
your own program, you will need to decide 
which recommendations you want to share only 
internally (e.g., if they have to do with staff or 
funding issues) and which ones will be shared in 
the final report and with your audiences/ 
stakeholders. 



Worksheets #7Aand 7B 



Identifying Criteria/Standards, Drawing 
Conclusions and Making Rec ommendations 

Make a copy of these worksheets for each member of.; 
the evaluation design team. As a group and based on 
your data analysis andinterpretations, use worksheet 
#7A to identify the standards against which you will base 
your judgments/conclusions. Using worksheet #7B, 
draw some conclusions based on your data analysis and 
indicate short- and long-term recommendations for 
program improvement. 




STEP 7 
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Worksheet # 7 B: Drawing Conclusions and Making Recommendations 
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Reporting Your 
Evaluation Results 



Getting the Message Out: Reporting 
Evaluation Information 




You have now identified your evaluation 
purpose and the research questions to guide 
the evaluation. You have employed various 
evaluation methods to collect data to address 
these questions. You have analyzed these data; 
and, based on this analysis, you have drawn 
conclusions about the program and have 
established recommendations for moving 
forward. Wow! Now what? Well, while all of 
these steps are crucial to an effective evaluation, 
your evaluation information will not be used by 
those who need it most, unless you take the 
final, and one of the most important, steps in 
the process: effectively communicating your 
results. 

Probably because it is one of the last steps, 
reporting evaluation information is often 
attended to superficially or with little attention 
to detail. Understanding all the aspects of 
reporting your evaluation information — from 
reporting the purpose oiyouv evaluation to 
tailoring the report to meet audience needs — 
will help to ensure that all of your hard work is 
put to good use. Given that, we present here an 
overview of some of the more important 
considerations in effective reporting. 



program worth based on resources consumed. 
Throughout this handbook, the focus has been 
placed on decision-making and as such, the 
evaluation report should provide information 
based on this purpose. In other words, your 
evaluation report should be framed in such a 
way that it will present information for making 
decisions about your program. There are, 
however, other purposes served by evaluation 
reports. A list of typical purposes includes the 
following (Worthen & Sanders, 1987): 

• To demonstrate accountability 

• To convince and gain support 

• To educate and promote understanding 

• To explore and investigate 

• To document 

• To promote public relations 

Identifying audiences for evaluation reports 

Before you can produce your evaluation report, 
you need to have a clear grasp of who the 
audiences are that will use the evaluation 
information contained within it. Without a 
clear understanding of the types of audiences the 
report targets, you could spend a lot of time 
creating a report that is neither understandable 
nor useful to its ultimate users. In addition, 
identifying your audiences will also help you to 
understand the types of questions they are likely 
to raise about the evaluation findings. In so 
doing, you can be proactive in addressing these 
issues in the report rather than simply reacting 
to them after the fact. 



Focusing the report on the evaluation’s purpose 

Step 1, which covered the purpose of 
conducting evaluation, stressed the importance of 
linking the evaluation to the role it was intended 
to play. For example, if the purpose of the 
evaluation was to guide program design and 
development, then the report should be used to 
inform staff and others involved in the design of 
the program about the adequacy of the design, 
reasonable alternatives, etc. If, on the other 
hand, the purpose of the evaluation was to make 
decisions about continuing the program, then 
the report could focus on determinants of 
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Creating and presenting the report 
based on audience information needs 

Obviously, your audiences may have different 
informational needs. For example, if staff from 
similar programs is one audience, they may seek 
information about the intricacies of the program 
and its activities. If another audience is funders, 
they may simply want the “bottom line” (e.g., 
did it do what it said it would do?). In preparing 
your evaluation report, you will need to consider 
the type of information your audiences will find 
more important, as well as the particular 
medium, format and style they are likely to 
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In writing the 
evaluation report, it is 
, wise to ask colleagues 
to review sections as 
you write them and to 
provide feedback before :ij 
completing the final §|§| 
report As you know, 
when one gets too close 
to something, it is easy 
to lose perspective. j 
'' Having someone whof jjjf 
was not directly 
■involved with the MMpji 
f evaluation read the Mil! 
§§report can provid llBlll li 
editorial support as well 
; as “reality checks" 
about things such as 
j interpretations, 

: conclusions, and 
recommendations. 
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appreciate. In other words, depending on the 
range of audiences and their information needs, 
you may need to develop a variety of 
presentation formats for your report. Often, 
however, you can accomplish this with one 
report. For example, you can provide a “user 
friendly” non-technical report that provides 
information in laymans terms but includes a 
technical appendix for those wishing to 
understand the details of the data collection 
process, as well as technical details about the 
data analysis procedures. You can also gain 
more mileage from your report by reformatting 
it to fit other audiences needs and interests. For 
example, for a community-wide dissemination 
strategy, you could use sections of the report to 
promote the program through a newspaper 
article or newsletter or through an interview on 
a local radio talk show. 

Evaluation information is communicated 
most commonly through written reports. It 
can, however, be presented and communicated 
through several other media, depending on what 
you think is the most effective way to reach your 
audiences. For example, if one audience is 
legislators (i.e., funders), who are often too busy 
to pore through a long report, you might choose 
to present the evaluation findings through 
dialogues or testimonies coupled with some 
succinct graphs and charts. Other examples for 
displaying evaluation information include: 

• Photo essays 

• Slide presentations 

• Film or videotape presentations 

• Computer presentations (e.g., putting 
it on the Internet) 

• Case studies and anecdotal portrayals 

• Test score summaries 

• Question & Answer sessions 

The choices are limited only by your 
imagination and creativity. Remember, 
regardless of the way in which you choose to 
report evaluation information, that information 
needs to be convincing and useful to your 
audiences. 
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Components of a good written evaluation report 

While you may tailor your written evaluation 
report depending on the unique context of your 
program, there are some generally agreed upon 
items that should be included in any evaluation 
report: 

An executive summary can be included within the 
full report or function as a stand-alone 
document (usually used with a large evaluation 
audience). The executive summary is just that, 
a succinct summary of the evaluation purpose, 
procedures and major findings and 
recommendations. The executive summary is 
usually 3 to 15 pages in length, again depending 
on the size and scope of the evaluation. 
Obviously, if your full evaluation report is only 
15 to 20 pages, your executive summary would 
be much shorter, about 3 or 4 pages. 

An introduction to the report serves to set 
the stage for the rest of the report. The 
introduction will focus on the purpose of the 
evaluation and describe the audiences the report 
is intended to serve. Often the introduction will 
address such questions as: Why was the 
evaluation conducted? What questions was it to 
address? Who is intended to use the evaluation 
information? Another purpose of the 
introduction is to talk about any limitations to 
the study that affected data collection, analysis, 
or interpretations. Lastly, the introduction can 
serve as a “guide” to the full report — oudining 
in brief the sections that follow. 

A focus of the evaluation section describes the 
object of the evaluation (i.e., the program) and 
places the evaluation within a context. This 
section also provides an overview of the 
evaluation questions used to focus the study. 

The evaluation plan and procedures section of the 
report gets down to the nuts-and-bolts of the 
evaluation design by providing an overview of 
the information collection plan, the data 
collection instruments used, and an overview of 
data analysis and interpretation. Remember 
your audience — you should present enough 
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detail about the process and procedures to 
ensure the credibility of your results without 
providing so much that your audiences are 
overwhelmed or disinterested. (You can always 
provide specific data collection methods and 
statistical procedures in a technical appendix.) 



The presentation of evaluation results section 
contains the results of the evaluation and 
provides the source for the subsequent section, 
conclusions and recommendations. This section 
should provide a complete summary, including 
illustrative charts and graphs, while leaving the 
more detailed data analyses to a technical 
appendix (if necessary). One convenient way to 
lay out your results is to organize your findings 
around your research questions or the goals of 
the evaluation. 



Conclusions and recommendations should be 
provided in their own section and should 
present the criteria and standards used to judge 
the program; the judgments about the program 
(e.g., the strengths and weaknesses of the 
program); and the recommendations for actions 
that might be taken over the short- and long- 
term for program improvement. 



Appendices might include detailed descriptions of 
evaluation procedures, detailed data tabulations 
or analyses, complete transcripts of focus groups 
or interviews, and other information that might 
be relevant to the full understanding of the 
evaluation but not appropriate for the body of 
the report. You can also include in the 
appendices the data collection instruments (e.g., 
surveys or interview protocols). 



We have presented here an overview of the 
key sections that comprise a written evaluation 
report. Following is a checklist of things that 
would typify most good evaluation reports 
(adapted from Worthen & Sanders, 1987): 

□ Reports are provided in time 
to be most useful 

D Report content is tailored 
to the audience(s) 

D Report format and style are tailored 
to the audience(s) 

□ An executive summary 

D An adequate introduction that sets 
the stage 

D Mention of any limitations to the study 

D Adequate presentation of evaluation 
plans and procedures 

D Effective organization of the 
presentation results 

□ All necessary technical information 
provided (preferably in the appendix) 

□ Specification of criteria and standards 
for evaluative judgments 

□ List of both strengths and weaknesses 

□ Recommendations for action 

□ Sensitivity to those affected by the 
evaluation findings 

□ Accurate and unbiased presentation 
of the results 

□ Effective communication through 
telling the story 

D Appropriate level of detail 

D Lack of technical jargon 

□ Use of correct, uncomplicated 
and interesting language 

□ Use of examples and illustrations 

□ Attention to visual appearance 
and appeal 



0 

ERI C - 

kij.«tas.ii^,i ecru jtjng New Teachers, Inc. 



86 



71 



Motes 



87 



ERIC 



72 



Measured Steps 





Mini-Case Studies of Program Evaluation 




In order to provide handbook readers with “real-life” examples of program evaluation, RNT 
invited several pre-collegiate teacher recruitment program directors to write mini-cases describing 
evaluations they had conducted and what they had learned from the process. Our criteria for 
selecting the practitioners included: 



• Program has a pre-collegiate teacher recruitment focus; 

•"Program director had conducted internal evaluation(s) and may have commissioned 
external evaluation(s); 

• Evaluation was based on a predetermined set of questions tied to program goals and 
objectives; 

• Evaluation included tracking of participants as well as examining the effectiveness of 
program components and features; 

• Multiple data collection methods were used; 

• Evaluation findings have been used to inform program improvements. 



In addition, we sought to provide mini-cases from a variety' of program types, ranging from 
single-site, local initiatives to statewide and regional multi-site efforts. All but one of the 
programs we selected are well-established and all are notable examples of school/college 
collaboration. 

Each mini-case begins with a brief overview of the program, continues with a description of 
the evaluation process and how it benefited the program, and concludes with challenges 
encountered and how they’ll be addressed in the future. Contact information is provided at the 
end of each piece, should you wish to contact the author directly for further information. 
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Consortium for Minorities in Teaching Careers 



Brief Overview 

The Consortium for Minorities in Teaching 
Careers was a collaboration among ten universities 
in five states and Puerto Rico to design and 
implement programs encouraging high school 
students to follow education and career paths 
leading to the teaching profession. It operated 
from 1990-1995 under two separate grants from 
the U.S. Department of Education totaling $2.5 
million. While the programs used a variety of 
models, they shared the objective of increasing the 
number of minority youngsters motivated to 
become teachers. The main activity for participants 
in the Consortium was hands-on teaching of 
young people. At most sites, participants received a 
number of training sessions in lesson planning, 
instructional methodology and child development 
principles. Under the guidance of lead teachers, 
they planned lessons and team-taught elementary 
school children. Close to 400 students participated 
across eight sites in Baltimore, New Orleans, Los 
Angeles, New York City (3), Wisconsin, and 
Puerto Rico (2). Given the geographical, ethnic 
and programmatic differences among the 
programs, establishment of a comprehensive and 
well-conceived evaluation plan was essential. 



Evaluation Process 

An independent evaluation unit at the Center for 
Evaluation arid Assessment within the College of 
Education at the University of Iowa designed and 
carried out formative and summative evaluation 
activities and developed instrumentation and data 
collection strategies that were implemented at each 
site. The Consortium site coordinators met 
annually at one of the program sites. In the early 
years, emphasis was on reviewing the data 
collection instruments, making modifications 
based on input from the site coordinators and 
training them in administrative procedures. In 
later years, emphasis was on interpreting results 
and making program modifications based on the 
lessons learned from the data collected. Interaction 
between evaluator and site coordinators was critical 



to the effectiveness of the evaluation. The 
evaluator was able to adjust procedures based on 
the realities at each site, while site coordinators 
gained confidence in the evaluation procedure 
through participation in design and modification. 

The evaluation plan included a series of 
process evaluation (formative) activities as well 
as outcome (summative) activities. Functions of 
the evaluation unit included: 

• development of protocols to guide the data 
collection activities of the on-site teams 

• development and refinement of all 
evaluation instrumentation 

• analysis and initial interpretation of all 
quantitative data 

• review, summarization, analysis and 
interpretation of qualitative data 

• management and coordination of 
evaluation information, including writing 
and producing evaluation reports 

• establishment through training, cooperation 
and consensus of a unified management 
information system (structured database) to 
facilitate standardized data collection 

• interpretation and dissemination of 
evaluation information about the projects 
at member institutions 

• consultation, including troubleshooting of 
evaluation problems for site evaluation 
teams 

• meta-evaluations of local site evaluation 
team activities in order to improve the 
evaluation process at individual sites and 
across the Consortium 

Functions of the on-site evaluation teams 
included: 

• implementation of data collection plans as 
stipulated in the Evaluation Design 
Workbook 

• administration of data collection 
instruments at the local site 

• conducting interviews and observations as 
stipulated in the final revised evaluation 
plan 
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• preparing initial analyses of all 
qualitative data 

• transmitting primary evaluation data and 
initial analyses to the evaluation unit 

• diagnosing problems with data collection 
or other evaluation problems that occur 
on site 

Process Evaluation The staff at each site 
monitored the functioning of local programs by 
documenting the activities of program staff and 
participants. They used both quantitative and 
qualitative methods, including: participant and 
staff surveys; participant interviews; direct 
observation and reporting of program events; 
demographic and background characteristics of 
participants, including numbers of participants in 
salient categories; and reviews and summaries of 
program documents, manuals, materials and reports. 

Questions that guided the design of the 
process evaluation were: 

• How are potential minority teacher 
candidates recruited and how can 
recruitment efforts be improved? 

• How can participation of recruited students 
be improved? 

• What are the most important characteristics 
of the pre-collegiate intervention programs 
and how can the programs be improved? 

• What program characteristics will affect 
dissemination of pre-collegiate intervention 
programs and how can dissemination 

be improved? 

Outcome Evaluation Via quantitative and 
qualitative methods, outcome evaluation 
addressed specific objectives for the pre- 
collegiate programs across all sites. 

Questions that guided the outcome 
evaluation were: 

• How can significant numbers of minority 
junior high and high school students be 
encouraged to enroll in intensive, 
pre-collegiate, future teacher preparation 
programs? 

• What changes occur in students attitudes 
toward children, the learning process and 
toward teaching as a viable career a 5 a result 
of participation in such programs? 



• What changes occur in students’ ability 
to work with children in a school 
environment as a result of participation 
in such programs? 

• What changes occur in students’ knowledge 
of and attitude toward critical subject 
matter and critical skills needed to become 
an effective teacher as a result of 
participation in such programs? 

• Do minority students enrolled in pre- 
collegiate programs encouraging entrance 
into teaching careers demonstrate academic 
improvement through higher grades, higher 
test scores, or qualifying for and selecting 
college preparatory classes as a result of 
participation in such programs? 

Reflections and Challenges 

Through the design oudined above, the evaluation 
model not only provided a framework for 
collecting and analyzing data to measure the 
success of the project, it also developed and 
increased the capacity of participants at each 
member institution to become more adept at both 
process and outcome evaluation activities during 
the course of the project. Overall, the surveys 
showed very strong changes in participants’ 
attitude about the status of the teaching profession 
and the interest in becoming teachers. In-depth 
interviews of selected samples confirmed the 
survey data and also indicated that the program 
caused many participants to have much more 
positive commitment to going to college regardless 
of the anticipated field of work after graduation. 
The direct interaction with young children in a 
teaching situation was an extremely positive 
experience for the great majority of classroom 
participants. 

Contact Information 

Dr. Joseph Braun 

Professor of Education 

California State University, Dominguez Hills 

1000 E. Victoria Street 

Carson, CA 90747 

Tel: (310) 243-2100 
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Future Teachers Program 



Brief Overview 

Future Teachers is a precollegiate teacher 
recruitment program whose initial purpose was 
to increase the diversity of students pursuing 
teaching careers. With the evolution of its two 
main programmatic strands, CIMATEC 
(Ciencia, Matematicia, and Technologia) and 
FTOMST (Future Teachers of Mathematics, 
Science, and Technology), Future Teachers now 
focuses specifically on the recruitment of 
teachers in mathematics, science, and 
technology. 

The Future Teachers Program has now been 
institutionalized in the Department of 
Secondary Education at California State 
University/Fullerton (CSU/Fullerton). The 
program has been in operation for 15 semesters 
and has served over 3,500 students who have 
provided over 120,000 service hours in 
elementary, middle, and high school tutoring. 
Many of these students are now in their first 
years of college at two- and four-year 
institutions. Each semester, over 250 new high 
school juniors and seniors complete the 
two-unit college course (“The Teaching 
Experience”) as part of the Future Teachers 
Program. 

The course is offered to California State 
University-eligible high school juniors and 
seniors in 18 high schools, and includes a 
weekly seminar, tutoring experience, career and 
academic advising, and a campus field trip with 
faculty presentations from the math and science 
departments. The university covers tuition, and 
students have access to the university library, 
health services, and other facilities while 
enrolled. High schools participating in the 
program were chosen because their districts 
serve as Professional Development Centers for 
the Single Subject Credential Program. 
Secondary teachers (one per participating 
school) who serve as Future Teachers Advisors 
receive a stipend and are considered adjunct, 



part-time faculty in the Department of 
Secondary Education. The department also 
provides three units of release time for a faculty 
member to direct the program. Additional 
grants have allowed us to pay stipends to a team 
of Faculty Advisors, who also attend seminars on 
the effective use of student journals; improving 
problem solving; writing in mathematics and 
science; developing students’ math and science 
skills; and other topics. Students gain experience 
tutoring in math/science-related subjects, and 
are asked to reflect on teaching in journal 
writing assignments. 

Evaluation Process 

Three categories of questions guided our 
program evaluation research. The first area of 
analysis dealt with career choice: 

• Does the Future Teachers experience 
increase positive attitudes toward 
teaching as a career choice? 

• Which components of the program are 
most influential on career decisions? 

• In what specific careers do students 
express most interest? 

The second area of analysis focused on 
whether the Future Teacher experience was able 
to address any of the reasons why students of 
color are not entering teaching. 

A third area of analysis addressed the 
tutoring experience: 

• In what ways did tutoring provide 
participants with an early field 
experience in teaching? 

• Did participants grow professionally as 
tutors? 

• Was it a positive experience for 
participants? 

• In what kinds of tutoring experiences 
did students participate? 
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Research methods included both qualitative 
and quantitative measures. Students completed a 
pre- and post-test questionnaire on the first and 
last days of the course and were surveyed on 
issues of program effectiveness. The pre-test/ 
post-test and end-of-program surveys were 
designed to determine attitude changes 
regarding teaching as a career choice as well as to 
identify effective program features. Tutoring 
logs, course journals, program documents, and 
newspaper articles on the program were 
qualitatively analyzed for common themes 
related to program evaluation. In addition, 
student journals were analyzed for common 
themes relating to personal histories, course 
experiences, and attitudes toward teaching. Two 
major findings pointed to the positive influence 
of the program. First, students identified the 
Future Teachers Program as the second greatest 
influence on their decision to pursue the career 
of teaching (“a teacher” was the primary 
influence). Second, participants identified the 
tutoring experience and the university field trip 
as the two most helpful activities. 

Reflections and Challenges 

Positive evaluation findings and the growing 
demand for teachers of mathematics, science, 
and technology prompted our decision to make 
the two strands — CIMATEC and FTOMST — 
the primary emphasis of the Future Teachers 
program. Consequently, we believe Future 
Teachers has begun to impact both the teaching 
of mathematics and science and recruitment of 
teachers in those subjects. However, we also 
learned through the monitoring process that all 
students were not receiving the same level of 
training. We responded to this challenge by 
increasing the frequency of faculty visits to 
schools, and by offering training workshops for 
advisors on such topics as: course development, 
arranging tutoring opportunities, increasing 
student awareness of campus facilities, and ideas 
for arranging valuable field trips and seminars. 

Student portfolios were yet another tool 
that enabled us to assess program effectiveness 
and monitor the progress of our future teachers. 
The portfolio fulfilled both process and product 
goals. As a process tool, the portfolio helped 



students integrate knowledge of, skills in, and 
attitudes toward teaching and learning; 
developed and refined reflective observation 
skills; and fostered professional orientations to 
problem solving, decision making, and 
leadership. As a product tool, the portfolio 
helped students develop a record of achievement 
that was richer and more illustrative than a 
transcript; chronicled self-growth; and 
promoted professional skills and abilities in the 
college market. The decision to include 
portfolios was also prompted by a variety of 
developments that have come in the wake of 
California's authentic assessment movement. For 
example, an area university (University of 
California/Irvine) recently used portfolios as the 
basis for student admissions. Furthermore, 
CSU/Fullerton teacher candidates, and first- 
and second-year teachers in induction programs, 
are all required to prepare portfolios. Therefore, 
adding portfolios at the high school level was 
consistent with requirements for other 
components of the pipeline into teaching. 

To others conducting internal evaluations 
we would suggest allotting at least one-third of 
your budget to evaluation. Our program grew 
very fast and we simply did not have the 
resources to conduct evaluation (internal and 
external) in the manner we felt necessary. We 
had originally wanted to follow up every student 
completing the program over a five-year period. 
The large numbers of students completing the 
program and student mobility of over 50 
percent, made this task impossible. Our current 
plan is to identify two students from each of the 
18 high schools to follow up over the next five 
years. 

Contact Information 

Dr. Helen Taylor 

Dr. Victoria Costa 

Future Teacher Program 

California State University/Fullerton 

Department of Secondary Education 

Fullerton, CA 92634 

Tel: (714) 278-3391 
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The Golden Apple Scholars of Illinois 



Brief Overview 

This nationally recognized program recruits, 
selects and prepares high potential, mostly 
minority (79%) and/or low-income high school 
graduates to teach in Illinois’ high need schools. 
Combining preservice training during four 
Chicago-area summer institutes with loan 
repayment, mentoring by award-winning 
teachers, university networking and work study 
opportunities during the school years, the 
Golden Apple Scholars garner approximately 
4,000-6,000 hours of training, workshops and 
classroom experience in urban schools prior to 
student teaching their senior year of college. In 
return, they commit to teach for five years in an 
Illinois high need school. 

The program began with 15 Scholars 
selected from the Chicago-area schools in 1989. 
By the summer of 1992, when the program 
expanded statewide, the Scholars numbered 
120, with 60 selected in the first statewide class 
of 1993 and the same number selected each year 
thereafter for a cohort of 240. The university 
network tripled from seven to 22 universities, all 
providing liaison and financial aid services to 
the Scholars and the foundation. 

In 1992, the programs budget was $50,000. 
The next year we received $80,000 planning 
grants from each of two state education 
agencies, as well as $50,000 from two city of 
Chicago offices. By 1996, the budget had grown 
to over $2 million. 

The first four classes of Scholars selected 
from 1989 through 1992 (n=62), were privately 
funded at $7500 each per year for four years. 
Therefore, the reporting requirements for this 
group included tracking several factors. The 
donors wanted to know the progress of each 
Scholar in school, his/her high school and 
community background, and retention and 
progress in the summer institutes. 

However, internal reportage and tracking of 
Scholar nominations (600-1,000 per year), 



retention, and descriptive statistics on the 
winners did not satisfy the state and city 
agencies. It was time to turn to external 
evaluation: to report outcomes, success stories, 
the “bang for the buck.” We also needed to 
examine our program goals and determine our 
next steps in program expansion. 

Evaluation Process 

At first, we thought we might evaluate the first 
Scholar graduates as they began their teaching. 
Since this cohort numbered 15> of whom only 
three were fully certified the first year after 
graduation, we decided to evaluate the 
relationship of the universities to the Scholar 
program. Universities in the downs tate area of 
Illinois, both public and private, had recently 
joined the network. Since universities were key 
to the recruitment, attendance, retention, and 
certification of Scholars, that became the nucleus 
of the first evaluation, “Expectations of 
Participating Universities.” We turned to the 
North Central Regional Educational Laboratory 
for help in determining the focus and parameters 
of the investigation. We answered three questions 
using scholar interviews, scholar surveys, 
telephone interviews with university liaisons, 
surveys with other university contacts, and staff 
interviews: 1) How well had the program been 
implemented at participating universities? 2) 
What implementation barriers occurred and how 
were they overcome? and 3) What impact did the 
program have on participating universities? 

With program costs spiraling, a second 
investigation, “An Exploratory Cost- Effect 
Analysis of the Golden Apple Scholars of 
Illinois Program” was undertaken as soon as 
the first evaluation was completed to determine 
the cost effect or value-added costs to educate 
Scholars versus traditional teacher candidates. In 
addition, each component of the program was 
examined and the total cost factored to come up 
with a percentage for each. The costs of each 
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segment were then analyzed to determine future 
programming and budget needs and clarify 
decisions about the original investment. 

Research questions used during the second 
evaluation included: 

• How well does the program do on the 
selected outcomes: 

a) graduation 

b) capability and “preparedness” of 
program teachers to teach in high need 
settings 

• What resources are involved in providing 
program services or activities? 

• What are the values of the resources used? 

• Who provides the resources — the project? 
teacher candidates? others? 

• What is the relationship of outcomes to 
costs for the Golden Apple Scholars 
program? 

The third evaluation, “Preparing Teachers for 
the Urban Classroom: An Examination of the 
Golden Apple Scholars of Illinois Program,” 
became the first phase of a longitudinal study of 
the Scholar Teachers who have graduated and are 
teaching in the Chicago Public Schools or Chicago 
area private schools (n=22 out of 25 graduates in 
1995). Human Capital Research Corporation, an 
education, policy, and research consulting firm, 
conducted our longitudinal study. 

Research questions used during the third 
evaluation included: 

• What are the characteristics of Scholars now 
teaching in urban settings? 

• What are the experiences and actions of 
teachers who have been prepared through 
this model program? 

• How long do they plan to stay in teaching? 

• What obstacles have they encountered in 
the field and what elements of support 
contribute to overcoming these obstacles? 
This third evaluation painted a portrait of 

the first-, second- or third-year Scholar teachers, 
documented the Scholar story and determined 
early program outcomes. As a baseline, we also 
surveyed more experienced teachers assigned to 
the same schools as the Scholar to learn what the 
teachers looked like who were teaching in the 
urban schools, how they built collaborative 



relationships, interacted with administrators, 
involved and communicated with parents, 
interacted with the community, experienced 
professional development activities, and 
addressed non-instructional classroom problems. 
Over 80% of the Scholars reported plans to stay 
in teaching, with 10% planning to leave 
teaching but stay in education, and 10% to 
teach in a different school or different grade. 

These results informed the fourth summer 
institute which was completely recast as a 
preservice teacher induction workshop designed by 
the Scholar teachers and the senior Scholars who 
had just completed student teaching. During the 
one week seminar, 52 Scholars met and developed 
their capacity to job search, prepare for student 
teaching, and got reacquainted with their cohort 
group (they attend over 20 separate universities). 

In addition, a formalized professional development 
series for Scholar teachers is being implemented. 
The role of external evaluation is crucial to 
determining progress, outcomes, and next steps. 

Reflections and Challenges 

It is always difficult to hear any criticisms of your 
program, but, pleasandy, by working closely with 
the evaluation team, what came out of this process 
was a much clearer understanding of what was 
working and why. Since we are funded by 
government agencies and the state legislature, it is 
much easier to explain our outcomes when backed 
by hard data, evaluated by a neutral but skilled 
party. Another realization is that you have a 
successful program when you hear the participants 
praise various aspects of it. 



Contact Information 

Dr. Victoria Davis 
Executive Director 
Illinois Learning Partnership 
203 West Hillside 
Naperville, IL 60540 
Tel: (630) 420-8427 
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The M & M Program 



Brief Overview 

The Minority Education and Recruitment into 
Teaching (MERIT) program at Springfield 
Technical Community College (STCC) was 
initiated in 1996 to increase the number of 
minority teachers for the Springfield, 
Massachusetts schools. MERIT is funded by 
the Minority Education Opportunity Program 
(MEOP), a state program to expand educational 
opportunities for minority students. At 
neighboring Springfield College (SC), the 
Mentors Inspiring New Directions (MIND) 
program, also funded by MEOP, was initiated in 
1996 to expose minority children to college and 
encourage them to attend. 

For the 1996-97 school year, the MERIT 
program and the MIND program combined to 
form the M & M Program. The partnership was 
inspired by the similarity of program goals and 
the potential for a well-rounded program for 
high school students when institutional 
resources were merged. Furthermore, the 
combined efforts of two colleges, one technical 
and one liberal arts, provided a unique 
opportunity for creative planning. 

Program Activities The program 
incorporates three main types of activity: 
academic enrichment, tutoring, team building, 
and field trips. Each week, faculty members 
from both colleges (STCC and SC) teach 
enrichment workshops in leadership, writing, 
computer math, and science. All of the after- 
school workshops are interactive and hands-on, 
with an emphasis on the teaching process. 
Professors begin workshops by describing their 
own journey into teaching, what schools they 
attended, and how their own teaching methods 
have evolved over the years. Once each week, 
six minority graduate students from SC and one 
undergraduate from STCC s minority teacher 
recruitment initiative with University of 
Massachusetts (Project STRIDE), serve as tutors 

0 



for individual or small groups of students after 
school. Mentor teachers are present to oversee 
the tutoring process. 

Between the academic enrichment and 
tutoring, students in the program have the 
opportunity to visit both campuses each week. 
Additional trips include a visit to the University 
of Massachusetts at Amherst, where students 
attend a college class in educational methods, 
visit the “observations corridor” at the campus 
elementary school, and tour the Campus Center 
to learn about the diverse student groups and 
activities on campus. Students also receive tours 
of both the SC and STCC campuses, and visits 
to other educational/cultural museums and 
exhibits. 

Evaluation Process 

The Planning Team Session, held at the 
beginning of the year, and the Team Evaluation 
Session, conducted at the end, were crucial to 
program planning and evaluation. Both were 
scheduled and established as a condition of 
employment in the contracts of all teachers, 
tutors, and mentors. 

Planning Team sessions served to prepare 
and orient the teachers, tutors, and mentors for 
the programs three main areas of focus: 
enrichment activities, academic tutoring, and 
team building. These preparatory meetings also 
gave tutors the opportunity to define their roles 
and get faculty input for structuring their 
tutoring sessions. 

At the Team Evaluation session, we 
discussed in-depth participants’ responses to 
selected questions from the end-of-year program 
evaluation survey form. Completed by all 
faculty and tutors prior to this meeting, 
evaluation forms included a common set of 
questions about the program, plus a page of 
questions specific to each participant (i.e., tutor, 
advisor, or instructor). Students were also asked 
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to complete a written evaluation and to discuss 
the questions in a session with their home 
school advisor. Program staff summarized major 
findings for each evaluation question and sent 
them out to all participants following the Team 
Evaluation session. 

As our program started later than planned 
and with fewer students than anticipated, on- 
going evaluation throughout the six months 
occurred informally, primarily via team meetings 
during which attendance data and observations 
about participation were shared. At the end-of- 
the-year Recognition Picnic for students and 
their families, parents engaged in informal 
discussion with the staff and most commented 
that their childrens grades had improved and 
motivation about school in general had 
increased. We saw these parental observations 
and the fact that about half of the students had 
perfect program attendance as positive program 
validation. 

Program staff also tracked student and 
program success by collecting statistics on the 
following: socio-economic factors (the 
percentage of students eligible for free or 
reduced lunch); ratio of male to female students; 
ethnic background; and academic performance 
(the percentage of students that showed 
increases and/or stability in their final grades in 
English, science, and/or math). 



Reflections and Challenges 

For the first year of the program, the evaluation 
process was quite sufficient. Though some 
participants thought the process was too 
extensive, all agreed that it was very meaningful 
to sit as a group and reflect on the program as a 
whole. Most importantly, we will implement 
some positive program changes and new 
initiatives for the following year based on the 
evaluation data. 

The evaluation questions will remain similar 
next year, but more statistical data will be 
collected from the participating schools in 
future years (particularly with regard to 
academic grades, attitude, and attendance upon 
entrance) to allow comparison in these areas 
from start to end of the program. Also, we will 
ask more individualized questions of students 
who continue in the program for a second year. 



Contact Information 

Ms. Sheila D. Blair 

Springfield Technical Community College 
1 Armory Square 
Springfield, MA 01101-9000 
Tel: (413) 781-7822 ext. 3198 
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Project PRIME, Programs to Recruit and Inspire 
Minorities in Education 



Brief Overview 

Morgan State University’s School of Education 
and Urban Studies (SEUS) is the lead 
institution for Project PRIME — Programs to 
Recruit and Inspire Minorities in Education — 
an initiative to increase the number of students 
of color who enter teaching careers. The DeWitt 
Wallace- Reader’s Digest Fund made a five-year, 
$3,008,000 grant to Morgan State University to 
support the project. 

PRIME is a collaborative effort among 
multiple programs to provide early recruitment 
and continuing support of students of color who 
enter teaching careers at partner institutions. 

In addition to Morgan State, the PRIME 
coalition includes the Baltimore City and 
Baltimore County Public Schools, the Maryland 
MESA Program, and several area colleges and 
universities. These institutions are committed to 
implementing a plan that will dramatically 
increase the number of African Americans in the 
Baltimore area who enroll in teacher education 
programs. Successful implementation of PRIME 
has spawned many precollegiate recruitment 
activities throughout the state of Maryland. 

Project goals are: 

1. To inspire minority youth, in particular 
African Americans, to consider teaching 
as a career; 

2. To enrich and enhance the image of 
teaching as a profession; 

3. To recruit minority youth, in particular 
African Americans, into mathematics 
and science teaching careers; 

4. To prepare teachers to work effectively 
with diverse student populations; 

5. To establish PRIME as a state resource 
center for minority teacher recruitment. 

PRIME includes a pipeline of eight 
programs aimed at recruiting and retaining 
students of color into teaching careers, 
beginning with interventions that start at the 



middle grade and high school levels — a time 
when students make course enrollment decisions 
that determine whether or not they will pursue a 
college preparatory path of study. Serving 
approximately 2,940 students annually, project 
activities reach students and families directly, 
informing them of (1) the need for a diversified 
teaching force, and (2) the “joys and challenges” 
of the teaching profession. Activities include the 
following: ProTeam and Teacher Prep classes, 
Future Educators of America chapters, campus 
tours, Citywide Program for the Teaching 
Professions magnet program, and the Hands-On 
Science for Elementary Students program 
(taught by middle and high school students). 

Evaluation Process 

Prior to 1995, PRIME engaged in formative 
evaluation activities, which included the 
following: participant response feedback forms, 
workshop evaluations, parent assessment forms, 
and narrative reflection activities. Students who 
received formal instruction through the 
ProTeam and Teacher Prep classes also 
completed pre/post surveys. At the end of each 
project year, the college partners engaged in a 
group self-study process. 

In May 1995, PRIME contracted with 
Research and Evaluation Associates, Inc. to 
conduct an external evaluation for project years 
four and five. The purposes of the evaluation 
were to “help implementers assess their progress 
and effectiveness; identify and target program 
components that need improvement; and 
inform others of the effectiveness and/or 
ineffectiveness of identified recruitment 
strategies.” 

Staff and mentors, in consultation with the 
evaluators, identified the following major 
questions: 

1. To what extent has PRIME contributed 
to students choosing teaching as a 
career? 
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2. What has been the effect of PRIMEs 
efforts to enrich and enhance the image 
of teaching as a profession? 

3. To what extent has PRIME been 
effective in recruiting ethnic minority 
youth into mathematics and science 
teaching careers? 

4. To what extent have efforts to enhance 
teacher effectiveness in culturally diverse 
classrooms been successful? 

5. To what extent has PRIME been 
established as a statewide resource 
center for precollegiate minority teacher 
recruitment? 

6. Does PRIME provide a pipeline from 
grades 6 through 12 to increase student 
interest in teaching? 

7. What has been the nature of 
collaborations between PRIME and 
higher education institutions? 

8. What are the unanticipated effects 
of PRIME? 

Two principal methods were utilized for 
data collection: interviews and surveys. 
Secondary sources included observations and 
document reviews. Using a pre/post test design, 
pre and post surveys were administered to all 
students participating in curriculum-based 
components delivered during the regular school 
day (ProTeam and Teacher Prep). Each class was 
matched with a control group. This same design 
was also used with PRIMETime with Clergy, 
the church-based component designed to 
enhance the image of the profession among 
congregants and to sensitize them to the need 
for African American teachers. We collected data 
from all PRIME component constituents and 
participants, including teachers, family 
members, and counselors, through the use of 
evaluation/feedback forms. Findings from the 
surveys, interviews, observations and document 



reviews were instructive and informative. They 
confirmed informal self-assessments relative to 
the effectiveness of curricula, the impact of 
school bureaucracy on effective implementation, 
and the importance of the affective domain in 
the lives of African American children. 

The evaluation process — especially internal 
evaluation — benefited project staff in several 
ways: immediate corrective feedback, data for 
long-range planning, realization of participant 
ownership in the project, clarification of project 
goals and objectives, and opportunities for mid- 
course corrections. 

Reflections and Challenges 

Problems with the evaluation process were 
inextricably tied to the challenges of effectively 
implementing an instructional program within 
an urban school district experiencing daily 
turmoil, political upheaval and state-imposed 
sanctions. We contended with matters of 
attrition, accountability, approval and 
availability. Additionally, communicating the 
significance of consistent and accurate 
documentation and data collection to teachers, 
counselors, principals, and others who deliver 
the curriculum and implement the strategies was 
an assiduous task. Because of the spontaneous 
nature of urban schools, timely data collection 
was particularly challenging. 



Contact Information 

Dr. Patricia Morris 

Dean, School of Education and Urban Studies 

Morgan State University 

Jenkins Building 308 

1700 East Cold Spring Lane 

Baltimore, MD 21251 

Tel: (410) 319-3385 
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South Carolina Center for Teacher Recruitment 



Brief Overview 

The purpose of the South Carolina Center for 
Teacher Recruitment (SCCTR) is to provide 
leadership in identifying, attracting, placing, 
and retaining well-qualified individuals for the 
teaching profession in our state. In addition, the 
SCCTR responds to changing needs for teachers 
from underrepresented populations, in critical 
subject fields, and in under-served geographical 
areas in South Carolina. When the SCCTR was 
created in 1986, the founders had the wisdom 
to insist on program evaluations and the 
collection of longitudinal data. Later, the 
governing bodies of the SCCTR implemented 
an external review of SCCTR that looked at all 
programs and operations and how they 
functioned in toto to address the states teacher 
recruitment needs. 1 

SCCTR offers a continuum of programs 
and services — all but one has undergone a 
formal evaluation. Most of the SCCTRs 
evaluation efforts and resources have gone to 
annual evaluations of the two largest 
programs — Teacher Cadet (2,300 students at 
145 sites, 30% minority) and ProTeam (800 
students at 38 sites, 64% minority). The 
College HelpLine Program and the Teacher Job 
Bank have also undergone formal evaluations 
and some data have been collected on the two 
summer institutes and the annual job fair 
(EXPO) sponsored by the Center. Evaluations 
utilized written surveys, focus groups, on-site 
and telephone interviews, and case studies from 
current student participants and teachers, 
administrators, college partners, former Cadets 
who are not in college, former Cadets who have 
become licensed to teach, former Cadets who 
did not become teachers, former Cadets who are 
currently teaching, parents, and Job Bank 
applicants. 



Evaluation Process 

Much of the information gleaned from recent 
evaluations of the Teacher Cadet and ProTeam 
Programs has been very similar to past 
evaluations and marginally helpful in improving 
programs. We have been challenged in recent 
years to move from evaluations that focused on 
inputs to ones that focused on results (e.g., how 
many former Cadets are teaching in South 
Carolina?) 

We have found that the questions you ask 
in designing an evaluation are critical in 
determining the quality and usefulness of 
information you receive from the study. Tying 
the questions to the mission of the program(s) is 
critical. Once the questions are determined, the 
next challenge is to determine the means (and 
indicators) of collecting the information within 
the resources that are allocated for evaluation. 
Currently, the SCCTR spends approximately 
two percent of its budget for evaluation and 
research — most recommendations state that ten 
percent is more appropriate. Currently, the 
following questions guide our programmatic 
evaluations: 

Teacher Cadet 

1 . Is the Teacher Cadet Program an effective 
tool for recruiting males and minorities 
into a career in education? 

2. Is the Teacher Cadet Program an effective 
tool for recruiting teachers in critical 
subject and geographic areas? 

3. Is the SAY (Science and Youth) component 
of the Teacher Cadet Program encouraging 
young people to consider teaching science? 

4. Is implementation of the Teacher Cadet 
Program, including recruitment initiatives, 
being carried out according to SCCTR 
guidelines? 



l.The external reviews were conducted by Dr. Russell French, University of Tennessee in 1991; Dr. Robert Shoenberg, University 
of Maryland in 1993, and Sally Bond, Horizon Research in 1996. For eight years, the annual program evaluations were conducted 
bv the South Carolina Educational Policy Center at the University of South Carolina. In 1997, the studies were conducted by Patricia 
lrns, Inc. 
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5. Do former Teacher Cadets become civic 
advocates for education? 

Pro Team 

1 . Does ProTeam increase the number of 
students wanting to go to college? 

2. Does ProTeam increase the number of 
students interested in considering teaching 
as a career option? 

3. Do the students elected to be in ProTeam 
possess the characteristics sought by the 
program? 

4. Is the program being implemented 
according to the guidelines established by 
SCCTR in terms of time allotted for the 
class, teacher and student section, and 
curriculum activities? 

5. To what extent are teachers who are 
selected for ProTeam experienced teachers 
with exemplary teaching skills and high 
student rapport? 

Longitudinal Tracking In addition to the 
information collected from current and past 
program participants, the staff collects and 
maintains a database on over 19,000 students. 
We begin tracking students when they 
participate in the ProTeam Program (middle 
school), and we have access to the teacher 
licensure database at the South Carolina 
Department of Education to determine how 
many become licensed, in what area(s) they are 
licensed, what college/university they graduated 
from, and whether or not they are teaching in 
geographic areas where there are shortages of 
teachers. The database indicates whether the 
student participated in the Teacher Cadet 
Program, if he/she received assistance through 
the College Help Line Program, and if he/she 



received a South Carolina Teacher Loan. The 
database also contains demographic information 
(race, gender, permanent address, current 
address, intended major in college, parents’ 
names, etc.). 

Reflections and Challenges 

Finding evaluators who are willing and have the 
expertise to conduct studies of teacher 
recruitment programs is often challenging. The 
evaluators need to understand the issues 
involved in precollegiate teacher recruitment. 
Producing reports that are succinct and readable 
by policy makers is another aspect of evaluation 
with which we have struggled. Most policy 
makers are not going to read 50-page reports; 
however, that depth is often needed by program 
administrators who use the results to make 
programmatic improvements. 

The most important lesson that we’ve 
learned from our experience is that you can’t 
afford not to have evaluations of your programs. 
Evaluations that show results will help you stay 
in business and get increases in funding. 



Contact Information 

Dr. Janice Poda 

South Carolina Center for Teacher Recruitment 

Canterbury House 

926 Oakland Avenue 

Winthrop University Station 

Rock Hill, SC 29733 

Tel: (800) 476-2387 
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£ Teacher World 



Brief Overview 



The Teacher World program seeks to motivate 
students to become creative teachers and 
leaders who believe all children, in a culturally 
diverse society, can learn at high levels and 
will make positive changes throughout the 
world. One of Teacher Worlds primary goals 
is to increase the percentage of minority 
teachers in the state of Wisconsin. Due to its 
success, the Teacher World program has 
become an established model for minority 
teacher recruitment in the state of Wisconsin. 



The Wisconsin Department of Public 
Instruction (DPI) began designing and 
implementing the program in 1989. In 
addition to primary financial support from 
the DPI, Teacher World receives support from 
the University of Wisconsin-LaCrosse, 
University of Wisconsin-Parkside, Milwaukee 
Public Schools, Wisconsin Education 
Association Council, Wisconsin Improvement 
Program, local teacher associations, and local 
school districts. Annual cost of the program 
is $80,000. 



Program Activities 

The program brings one hundred students — 
fifty percent of whom are students of color — 
to spend one week during the summer on the 
UW-LaCrosse campus. Participants are 
selected from a pool of public and private 
high school applicants from across the state. 

During the program, students: plan and 
teach actual lessons; discuss current issues; 
listen to experiences of various professional 
educators; participate in a college recruitment 
fair; and take special classes in art, drama, 
music, and/or dance. The students also 
receive academic counseling; learn what is 
required for admission to Wisconsin’s best 
teacher training programs; and discover the 
programs that best suit their needs. 



Evaluation Process 

Students complete a detailed on-site 
evaluation on the final morning of the 
program in which they rate each program 
activity. Data from these evaluations are used 
by teachers and staff at both the fall and 
spring meetings to refine and/or revise the 
program in preparation for the summer. 

In addition, a five-year follow-up study 
was conducted using a random selection of 25 
percent of the students who attended the 
1989-1993 Teacher World Programs. The 
five-year benchmark was chosen in order to 
coincide with students’ junior year in college, 
when it is presumed that they would have 
elected, and begun work in, a major course of 
study such as education. The results indicate 
that 85-90 percent of the students enrolled in 
college, and approximately 50 percent of 
those students are enrolled in schools/colleges 
of education preparing for a career in 
teaching. 

To increase the likelihood that students 
would be available for comment, all calls were 
made to home phone numbers during the 
December holiday break. This strategy was 
largely successful, and the feedback resulting 
from students greatly enriched the evaluation 
reports. 



Reflections and Challenges 

Because the Teacher World director acts to 
facilitate, rather than to dictate, the course of 
the program, the corps of teachers who work 
in the program each year are empowered to 
make decisions regarding program 
development, revision, and implementation. 
The student evaluations have been 
instrumental in supporting program 
development and improvement because data 
(both qualitative and quantitative) from them 
are examined and discussed during bi-annual 
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planning sessions. During these fall and 
spring weekend retreats, the master teachers 
(permanent Teacher World faculty) and 
teaching assistants (college students) are 
brought together to plan for the coming 
summer. 



Contact Information 

Mr. James R. Wickman 

Bureau for Teacher Education, Licensing 

and Placement 

Wisconsin Department of Public Instruction 
125 S. Webster Street 
Madison, Wl 53707-7841 
Tel: (608) 266-9352 
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0 Walton/Lehman Pre-Teaching Academy 



Brief Overview 

The goals of the program are to give qualified 
minority teenagers first-hand experience with 
teaching, provide them with the support and 
training necessary for success as both students 
and teachers, and motivate them to enter the 
teaching profession. Unanticipated outcomes 
have led to additional goals: re-energizing 
teachers who serve as mentors; increasing 
student outcomes in classes to which interns are 
assigned; and improving the school climate. 

Program objectives for pre-teaching 
students are to: promote positive attitudes 
toward teaching; improve self-confidence and 
presentation skills; provide first-hand experience 
in teaching; improve academic skills; ensure 
college readiness; and provide accelerated 
education through exposure to college courses. 
Objectives for teachers are to: increase morale 
and motivation; increase collegial 
communication; provide professional 
development; and disseminate teaching 
methods. Objectives for students in classes 
where pre-teachers are assigned are to: provide 
interaction with positive peer role models; 
increase individualized attention; and encourage 
interest in the teaching profession. 

Program OperationThe Pre-Teaching 
Academy was established in 1984 1 with a 
budget of $60,000 and $50,000 in in kind 
contributions (teacher time and college credits), 
and continues as a partnership between Lehman 
College and Walton High School. The 
academy’s philosophy is that the sooner students 
can be “turned on” to the joys and satisfactions 
of teaching, the more likely they will be to make 
an informed, realistic choice about teaching as a 
career. During the school year and in the 



summer, approximately 75 high school juniors 
and seniors (99% of whom are Latino, African 
American, or Asian) complete teaching 
internships under the supervision of high school 
teacher- mentors. Through classroom 
observation of their mentors and their own 
forays into teaching, they actually experience 
what works and what doesn’t. They witness first 
hand the application of theory in practice. 

Pre-teachers have a wide range of classroom 
experiences during their four semesters and two 
summers in the program: tutoring, instructing 
small groups, reviewing homework, presenting 
preliminary lesson materials, teaching parts of 
lessons, teaming with their men tor- teachers to 
teach entire lessons, teaching entire lessons on 
their own and preparing and grading tests. Pre- 
teachers have also taught entire lessons in classes 
in which in-subject substitute teachers were not 
available. 

Pre-teachers are registered for one credit of 
“p re- teaching” in each of their four internship 
terms. They receive a grade for the course 
determined by their cooperating teacher in 
conjunction with the program coordinator. Part 
of their responsibility is to keep a journal and a 
portfolio of lessons; these are reviewed by the 
program coordinator. 

All interns complete a three-credit course, 
“Introduction to Education,” 2 offered on the 
Lehman campus, and most go on to take an 
additional college course, “Introductory 
Speech.” 

Evaluation Process 

Lehman College conducts formative and 
summative evaluations of the program via the 
following methods: program evaluation surveys 





0 



1 Past funding for the programs has come from the Department of Education Fund for Innovation in Education, Aaron Diamond 
Foundation, and the New York Alliance for Public Schools’ “Mentor in Education” program. 

2 The text used in the Introduction to Education course is a 319-page manual written collaboratively by pre-teachers and mentors for 
use in the program. It includes such topics as: The Art of Questioning, Handling Problem Situations, Using Alternative Learning 
c trategies (with sample lessons in a variety of subject areas), and Planning Effective Lessons. 
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completed by pre-teachers and mentor-teachers; 
written teaching observations of interns; review of 
student excerpts over the four internship semesters 
from p re- teacher journals with publication in a 
monthly program newsletter, Chalk Talk; 
occasional formal pre-teacher evaluation by 
students in assigned classes; anecdotal reports on 
interns by their mentors; review of lesson 
portfolios; review of school records; review of 
standardized test scores; tracking (when possible) 
of college entrance, graduation and entry into 
teaching. An additional aspect of the evaluation is 
to solicit participant suggestions for program 
improvement. 

Key Questions the Academy is 

Trying to Answer: 

1) Does the academy promote positive 
attitudes towards the field of education? 

2) Does the academy encourage young people 
to become teachers? 

3) Does the academy prepare students for 
college? 

4) Does the academy provide students the 
opportunity to accelerate their education? 

5) Does the academy expose students to a 
college environment? 

The on-going evaluation has consistently 
demonstrated positive outcomes. The Academy: 

a. Promotes positive attitudes toward the field 
of education in both interns and 
mentors, as evidenced by survey results, 
journal entries, and intent of interns to 
enter college programs to prepare for 
teaching. 

b. Encourages young people to become 
teachers as evidenced by the 23% of 
(800+) pre-teaching participants since 
1984 who have entered careers in teaching. 
Many current pre-teachers indicate that 
their interest was sparked through having a 
peer as a pre-teacher in their class. 

c. Prepares students for college entry as 
evidenced by the 95% of interns who 
graduate from high school within four years 
and enter college. 



d. Provides interns the opportunity to 
accelerate their education as evidenced by 
the 100% of interns who earn at least three 
college credits; the 75% who earn at least 
six credits; and the 20% who earn nine or 
more college credits. 

e. Exposes students to a college environment 
as evidenced by the fact that interns and 
mentors frequently attend workshops, 
courses and events at the college. 

f. Improves student self-confidence as 
evidenced by survey results, by 
changing journal entries, and by anecdotal 
reports of mentor teachers. 

g. Reduces teacher isolation as evidenced by 
mentors' anecdotal reports, survey 
results, participation levels in workshops, 
and collegial tone of the school. 

h. Exposes interns and mentor teachers to 
new teaching methods and strategies as 
evidenced by: comments in journal entries; 
survey results; observation of the 
dissemination patterns of new/innovative/ 
creative teaching methods, strategies and 
techniques. The interns are, in a very real 
sense, like bees who carry these 
methods, strategies and techniques from 
internship assignment to internship 
assignment trying them out in the 
classroom and spreading them to new 
teacher mentors. This has proved a far 
more effective (though perhaps not 
efficient) dissemination technique than 
holding workshops. 

Evaluating the program enabled us to 
determine what was working well and what needed 
modification. We were also able to determine how 
different constituencies within the school and the 
* college perceived the program. 3 With this 
information we were able to strengthen the 
program and improve outcomes. We were also able 
to submit the program to an external evaluation 
process which resulted in program validation 
by the New York State Education 
Department. Without ongoing evaluation this 
would have been impossible. At present we 
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y These included: program alumni, administrators, teachers serving as mentors, other teachers in the school, interns and students in 
whose classes pre-teachers served. 
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have been given the opportunity to apply for 
program replication funds. Without the data 
we gathered over the years, further funding, 
especially for replication, would be difficult, if 
not impossible, to obtain. 



Reflections and Challenges 

We have encountered several challenges when 
evaluating: 

• Tracking former students to: find out what 
they were currently doing; ascertain long- 
term program impact on them; get input 
about what they saw as strengths and 
weaknesses of the program over time; and 
ask what they would change and what they 
would maintain. 




• Finding appropriate equivalent measures to 
use with a multilingual population of 
students served by preteachers. 

• Finding ways to ascertain some of the more 
elusive and hard-to-measure changes that 
interns experience, e.g., increased 
self-confidence, enhanced self-esteem, 
improved ability to present to large groups, 
and attitude toward teachers, education and 
school. 



If I were to begin this program in a new 
environment, a different school, knowing what I 
know now, I would spend more time initially 
collecting baseline data on: school climate; 
teacher self-perception; teacher perception of 
students; academic achievement for potential 
interns and for students they will pre-teach. I 
would also find ways to do content analysis of 
student logs and journals. Finally, I would 
videotape students teaching from early in their 
internships as a record of their growth and 
would have each intern maintain a portfolio of 
lesson plans, essays, class work, etc. 



Contact Information 

Dr. Anne Rothstein 
Associate Provost 
Lehman College 

250 Bedford Park Boulevard West 
Bronx, NY 10468 
Tel: (914) 960-8569 
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More Tips From Practitioners 



Getting Started 

%/ It is never too soon to start the evaluation. 

%/ Remember that no amount of hard work 
can make up for the lack of planning on the 
front end. 

%/ View evaluation as a necessary and critical 
part of the success of your program. 

%/ Develop and/or review a mission statement 
(purpose) and objectives (what you hope to 
accomplish) for your program. 

%/ Base your evaluation on your mission 
and objectives. 

%/ Be explicit about what you will evaluate 
and why; determine questions you want 
to answer. 

%/ Consider how you will answer the questions 
(e.g., methodology: surveys, interviews, data 
retrieval, etc.). 

%/ Consider how data will be used and how 
results will be reported. 

%/ Ask: what story do I want to tell? 

%/ Define what success means for your program. 
%/ Write evaluation into grant proposal 
budgets — funders like to see that you’re 
planning to evaluate. 

%/ Determine the audiences for your evaluation 
(e.g., policy makers, legislators, state board, 
donors, your board of trustees, program staff 
and participants). 

l/ Bring together as many stakeholders/ 

representatives of your audience(s) as possible 
to: discuss evaluation, create buy-in, and get 
multiple perspectives on your program. 

%/ Find out what is important to your policy 
makers and build those components into 
your evaluation. 

%/ Establish a pace (when and how often to 
collect data). 

%/ Be prepared to work through all the “red 
tape” of collecting data (e.g., students right to 
privacy, parental permission). 

For external evaluation: develop a 
professional relationship with the program 
evaluator(s) so that he/she will feel comfortable 
asking for additional information and you’ll feel 
comfortable offering suggestions; remember that 
the evaluator you choose must have credibility 
and be respected by policy makers. 

rtf 



Data Collection 

%/ Include qualitative information, if possible 
(let the “stories” tell your story). 

%/ Explore all possible sources of data (e.g., state 
department teacher licensure databases, loan 
recipients). 

%/ Describe what your program is about 
(for descriptive part of evaluation); convey 
the essence of the program (mission, goals, 
objectives, core values of program). 

\/ Document your projects history. 

%/ Track your own indicators: school retention, 
program retention, grade point average, 
honors, ethnicity, gender, etc. 

s/ Systematically collect data and organize it 
in a manner that is easy to retrieve. 

%/ Continuously communicate to students, 
teachers, administrators, etc. of the purpose 
of ongoing evaluations and the importance 
of their participation. 



Reporting Your Findings 

*/ Determine who should receive a copy of your 
evaluation results and make sure they receive 
a succinct report that is understandable in 
laymans terms. 

%/ Explain how you plan to implement the 
recommendations. 



Long-range Uses and Benefits 

\/ Listen to your critics (no matter how 
painful) — they may have some excellent 
suggestions for ways to improve. 

%/ Annually re-evaluate your evaluation plan 
and make necessary modifications. 

%/ Use recommendations from evaluations to 
assess current state of program, make 
program improvements, develop goals for the 
upcoming year. 

%/ Don’t throw away data that have been 
collected; you may not use the data now, 
but the information could come in handy 
later. 
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0 Glossary* 



Anonymity (provision for): Evaluator action to 
ensure that the identity of subjects cannot be 
ascertained during the course of a study, in 
study reports, or in any other way. 



Cohort: A term used to designate one group 
among many in a study. For example, “the first 
cohort” may be the first group to have 
participated in a training program. 



Assessment: Often used as a synonym for 
evaluation. The term is sometimes 
recommended for restriction to processes that 
are focussed on quantitative and/or testing 
approaches. 

Attrition: Loss of subjects from the defined 
sample during the course of a longitudinal 
study. 

Audience(s): Consumers of the evaluation; those 
who will or should read or hear of the 
evaluation, either during or at the end of the 
evaluation process. Includes those persons who 
will be guided by the evaluation in making 
decisions and all others who have a stake in the 
evaluation (see stakeholders). 

Background: The contextual information that 
describes the reasons for the project, its goals, 
objectives, and stakeholders’ information needs. 

Baseline: Facts about the condition or 
performance of subjects prior to treatment or 
intervention. 

Case Study: An intensive, detailed description 
and analysis of a single project, program, or 
instructional material in the context of its 
environment. 



Comparison group: A group that provides a basis 
for contrast with (in experimentation) an 
experimental group (i.e., the group of people 
participating in the program or project being 
evaluated). The comparison group is not 
subjected to the treatment (independent 
variable), thus creating a means for comparison 
with the experimental group that does receive 
the treatment. Comparison groups should be 
“comparable” to the treatment group, but can be 
used when close matching is not possible (see 
also Control Group). 

Content analysis: A process of systematically 
determining the characteristics of a body of 
material or practices. 

Control group: A group that does not receive the 
treatment (service or product). The function of 
the control group is to determine the extent to 
which the same effect occurs without the 
treatment. The control group must be closely 
matched to the experimental group. 

Correlation: A statistical measure of the degree of 
relationship between variables. 

Cost-effectiveness: This analysis determines what 
a program or procedure costs against what it 
does (effectiveness). Is this product or program 
worth its costs? 



Coding: To translate a given set of data or items 
into machine- readable categories. 




Criterion, Criteria: A criterion (variable) is 
whatever is used to measure as success, e.g., 
grade point average. 



* Excerpted from: Frechtling, J., Stevens, F., Lawrenz, F., Sharp, L., (1993). User-Friendly Handbook for Project Evaluation: Science , 
Mathematics, Engineering and Technology Education . Washington, D.C.: National Science Foundation. 



ERIC 



necruiting New Teachers, Inc. 



1 1 0 BEST COPY AVAILABLE 



95 



Descriptive statistics: Those that involve 
summarizing, tabulating, organizing and 
graphing data for the purpose of describing 
objects or individuals that have been measured 
or observed. 

Design: The process of stipulating the 
investigatory procedures to be followed in doing 
a certain evaluation. 

Dissemination: The process of communicating 
information to specific audiences for the 
purpose of extending knowledge and, in some 
cases, with a view to modifying policies and 
practices. 

Effectiveness: Refers to the conclusion of a Goal 
Achievement Evaluation. “Success” is its rough 
equivalent. 

Executive summary: A non-technical summary 
statement designed to provide a quick overview 
of the full-length report on which it is based. 

External evaluation: Evaluation conducted by an 
evaluator from outside the organization within 
which the object of the study is housed. 

FOCUS group: A group selected for its relevance to 
an evaluation that is engaged by a trained 
facilitator in a series of discussions designed for 
sharing insights, ideas, and observations on a 
topic of concern. 

Formative evaluation: Evaluation designed and 
used to improve an intervention, especially 
when it is still being developed. 

Generalizability: The extent to which 
information about a program, project, or 
instructional material collected in one setting 
can be used to reach a valid judgment about 
how it will perform in other settings. 

Impact evaluation: An evaluation focussed on 
outcomes or pay-off. 



Implementation evaluation: Assessing program 
delivery (a subset of Formative evaluation). 

Indicator: A factor, variable, or observation that 
is empirically connected with the criterion 
variable, a correlate. For example, judgment by 
students that a course has been valuable to them 
for pre-professional training is an indicator of 
that value. 

Inferential statistics: These statistics are inferred 
from characteristics of sample to characteristics 
of the population from which the sample comes. 

Informed consent: Agreement by the participants 
in an evaluation of the use of their names and/or 
confidential information supplied by them in 
specified ways, for stated purposes, and in light 
of possible consequences prior to the collection 
and/or release of this information in evaluation 
reports. 

Instrument: An assessment device (test, 
questionnaire, protocol, etc.) adopted, adapted, 
or constructed for the purpose of the evaluation. 

Internal evaluator: Internal evaluations are those 
done by project staff even if they are special 
evaluation staff, that is, external to the 
production/writing/teaching/service part of the 
project. 

Longitudinal Study: An investigation or study in 
which a particular individual or group of 
individuals is followed over a substantial period 
of time to discover changes due to the influence 
of the treatment, or maturation, or 
environment. 

Matrix: An arrangement of rows and columns 
used to display components of evaluation 
design. 

Mean: Also called “average” or arithmetic average. 
For a collection of raw test scores, the mean score is 
obtained by adding all scores and dividing by 
the number of people taking the test. 
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Measurement: Determination of the magnitude 
of a quantity. 



Median: The point in a distribution which 
divides the group into two, as nearly as possible. 
For example, in a score distribution, half of the 
scores fall above the median and half fall below. 



Mode: The value which occurs more often than 
any other. If all scores (in a score distribution) 
occur with the same frequency, there is no 
mode. If the two highest score values occur 
with the same frequency, there are two modes. 

Needs assessment: Using a diagnostic definition, 
need is anything essential for a satisfactory mode 
of existence or level of performance. The 
essential point of a needs assessment for 
evaluation is the identification of performance 
needs. 



Norm: A single value, or a distribution of values, 
constituting the typical performance of a given 
group. 



Objective: A specific description of an intended 
outcome. 



Observation: The process of direct sensory 
inspection involving trained observers. 

Outcome: Post-treatment or post-intervention 
effects. 



Peer review: Evaluation done by a panel of 
judges with qualifications approximating those 
of the author or candidate. 



Performance-based: The use of global ratings of 
behavior assessment which is a movement away 
from paper-and-pencil testing. This assessment 
is costly and there may be a loss of validity and 
reliability. 
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Pilot test: A brief and simplified preliminary 
study designed to try out methods to learn 
whether a proposed project or program seems 
likely to yield valuable results. 

Planning evaluation: Evaluation planning is 
necessary before a program begins, both to get 
baseline data, and to evaluate the program plan, 
at least for evaluability. Planning avoids 
designing a program that is unevaluable. 



Population: All persons in a particular group. 



Post test: A test to determine performance after 
the administration of a program, project, or 
instructional material. 

Pretest: A test to determine performance prior 
to the administration of a program, project, or 
instructional material. Pretests serve two 
purposes: diagnostic and baseline. Also the use 
of an instrument (questionnaire, test, 
observation schedule) with a small group to 
detect need for revisions. 



Process evaluation: Refers to the evaluation of 
the treatment or intervention. It focuses entirely 
on the variables between input and output. 

Product: A pedagogical process or material 
coming from research and development. 

Progress evaluation: A subset of formative 
evaluation. 

Prompt: Reminders used by interviewers to 
obtain complete answers. 

Qualitative evaluation: The part of the evaluation 
that is primarily descriptive and interpretative, 
and may or may not lend itself to quantitative 
treatment. 

Qualitative evaluation: An approach involving the 
use of numerical measurement and data analysis 
based on statistical methods. 
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Random: Affected by chance. 

Random sampling: Drawing a number of items of 
any sort from a larger group or population so 
that every individual item has a specified 
probability of being chosen. 

Reliability: Statistical reliability is the consistency 
of the readings from a scientific instrument or 
human judge. 

Sample: A part of a population. 

Self-administered instrument: A questionnaire or 
report completed by a study participant without 
the assistance of an interviewer. 

Self-report instrument: A device in which persons 
make and report judgments about the 
performance of their project, program, or 
instructional material. 

Significance: Overall significance represents the 
total synthesis of all you have learned about the 
merit or worth of the program or project. This 
is different from statistical significance which 
may be testing one of several conditions of a 
program or project. 



Stakeholder: A programs stakeholder is one who 
has credibility, power, or other capital invested 
in the project, and thus can be held to be to 
some degree at risk with it. 

Summative evaluation: Evaluation designed to 
present conclusions about the merit or worth of 
an intervention and recommendations about 
whether it should be retained, altered, or 
eliminated. 

Unanticipated outcomes: A result of a program or 
interview that was unexpected. Often used as a 
synonym for side-effects, but only a loose 
equivalent. 

Utility: The extent to which an evaluation 
produces and disseminates reports that inform 
relevant audiences and have beneficial impact on 
their work. 

Validity: The soundness of the use and 
interpretation of a measure. 
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0 Resources for Program Evaluation 



On-Line Resources 



Fund for the Improvement of Postsecondary 
Education 

Notes on Evaluation Design. 
http://www.ed.gov/offices/OPE/FIPSE/notes.html 

National School Network 

NSN Resource Exchange: Standards, Evaluation and 
Assessment Desk 

http://www.nsn.bbn.com/resources/desks/evaluation/ 

index.shtml 

Includes information on program and project 
evaluation, national and state standards, other 
evaluation rubrics, student and teacher 
assessment 

American Evaluation Association 

http://www.eval.org/ 

CRESST: National Center for Research on 
Evaluation, Standards, and Student Testing 

http://www.cresst96.cse.ucla.edu/index.htm 
Funded by the U.S. Department of Education, 
CRESST conducts research on important topics 
related to K-12 educational testing. Register for 
free newsletters about the latest CRESST 
research, and related links page for other web 
sites with valuable assessment information. 



Books on Evaluation 



Babbie, E. (1990). Survey Research Design. Belmont, 
CA: Wadsworth Publishing Company. 

Brinkerhoff, R., Brethower, D., Hluchyj, T., and 
Nowakowski, J. (1983). Program Evaluation: A 
Practitioner’s Guide for Trainers and Educators. 
Boston: Kluwer-Hijhoff. 

Cronbach, LJ. (1982). Designing Evaluations of 
Educational and Social Programs. San Francisco: 
Jossey-Bass Publishers. 

Frechtling, J., Stevens, F., Lawrenz, F., Sharp, L. 
(1993). User-Friendly Handbook for Project 
Evaluation: Science, Mathematics, Engineering, and 
Technology Education. Washington D.C.: National 
Science Foundation. 

Can be accessed on the internet at: http://www. 
ehr.nsf.gov/EHRIRED/EVAI_/handbook.html 

Herman, J.L., ed. (1987). Program Evaluation Kit. 
Newbury Park, CA: Sage Publications. 

Sanders, J.R., Chair, Joint Committee on Standards 
for Educational Evaluation. (1994). The Program 
Evaluation Standards, 2 nd ed., How to Assess 
Evaluations of Educational Programs. Thousand 
Oaks, CA: Sage Publications. 



ERIC Clearinghouse on Assessment and Evaluation 

http://ericae.net 

Question/Problem: Qualitative and Quantitative 
Research Methods page includes: responses to 
questions such as “What distinguishes qualitative 
research from quantitative research? When is it 
appropriate to use each type?”; other internet 
resources; non-ERIC bibliographic resources; 
ERIC Digests; ERIC citations and abstracts; 
instructions for ERIC Documents Access. 
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Weiss, C. (1997). Evaluation.- Methods for Studying 
Programs and Policies. Upper Saddle River, NH: 
Prentice-Hall. 

Wholey, J.S., Hatry, H.P., Newcomer, K.E., eds. (1994). 
Handbook of Practical Program Evaluation. 

San Francisco: Jossey-Bass Publishers. 

Worthen, B. and Sanders, J. (1988). Educational 
Evaluation: Alternative Approaches and Practical 
Guidelines. New York: Longman. 







The Grantseeker's Guide to Project Evaluation. Effective Evaluation: A Systematic Approach for 

(1997). Alexandria, VA: Capitol Publications, Inc. Grantseekers and Project Managers. (1993). 

Alexandria, VA: Capitol Publications, Inc. 
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Measured Steps: Reader Response Form 

We welcome your comments on this handbook. What you tell us will help us to revise and improve Measured Steps for future use. Please return your completed form by 
FAX to Recruiting New Teachers , Inc., at (617) 489-6005 or by MAIL to Recruiting New Teachers, Inc., 385 Concord Avenue, Suite 103, Belmont, MA 02478. Thank you! 



Name 



Title 



Affiliation 



Phone 



Address 



1 . How have you used this handbook to date? 

□ I have read it, but have not used it in conducting an evaluation as yet — Skip to # 4 . 



□ I have used it in the evaluation of (specify program, activity, etc.): 



2. How would you rate your evaluation skills before you used this handbook ( check one only): 

□ None □ Basic O Moderate □ High 

3. How would you rate your evaluation skills after you used this handbook ( check one only): 

□ None □ Basic □ Moderate □ High 

4. Please rate the following aspects of the handbook with an “1” being Excellent and an “5” being Poor ( circle one only): 

a. The purpose and objectives of the handbook are clearly presented. 1 2 3 4 5 

b. The handbook is well organized. 1 2 3 4 5 

c. The content is at an appropriate level for the novice. 1 2 3 4 5 

d. The handbook prepared me to apply what was learned. 1 2 3 4 5 

5. Please identify the most valuable feature(s) of this handbook for you: 



6. Did your understanding of evaluation and evaluation methods change as a result of your use of this handbook? 
□ YES □ NO Please explain: 



7. Was there something you had hoped to learn from this handbook, but did not? Please explain: 



Please feel free to provide additional comments on the reverse side of this form. (If you FAX this back to Recruiting New 
Teachers, Inc., please make sure to make a copy of any comments on the reverse side.) 
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